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PREFACE 


This  note  documents  a  study  conducted  jointly  under  Task  Order 
80-V-I,  Survey  Research,  and  Task  Order  80-I11-I,  Reserve  Forces 
Manpower,  as  part  of  Rand's  Manpower,  Mobilization,  and  Readiness 
Program,  sponsored  by  the  Office  of  the  Assistant  Secretary  of  Defense 
(Manpower,  Reserve  Affairs  and  Logistics) — OASD(MRA&L). 

kith  manpower  issues  assuming  an  ever  greater  importance  in  defense 
planning  and  budgeting,  the  Rand  study  program  seeks  to  develop  broad 
strategies  and  specific  solutions  for  dealing  with  present  and  future 
defense  manpower  problems.  The  achievement  of  these  goals  requires  the 
development  of  new  methodologies  for  examining  broad  classes  of  manpower 
problems,  as  well  as  specific  problem-oriented  research.  In  addition  to 
analyzing  current  and  future  manpower  issues,  the  study  program  seeks  to 
contribute  to  a  better  general  understanding  of  the  manpower  problems 
confronting  the  Department  of  Defense. 

A  key  component  of  the  program  is  the  development  of  DoD-wide  data 
bases  to  support  the  policy  formulation  and  research  necessary  for 
dealing  with  present  and  future  defense  manpower  problems.  Such  data 
should  include  information  about  the  behavior,  experiences,  attitudes, 
preferences,  and  intentions  of  military  personnel.  Particularly  if 
collected  on  a  periodic  basis,  these  data  could  be  used  to  assess  the 
response  of  military  personnel  to  past  and  current  policy  changes  and  to 
identify  future  areas  for  policy  action. 

The  1979  Reserve  Force  Studies  Surveys  are  part  of  several 
interrelated  data  collection  efforts  of  the  Rand-DoD  Survey  Group,  a 
component  of  the  Rand  Manpower,  Mobilization,  and  Readiness  Program.* 

The  survey  group  has  designed  and  administered  two  other  military  life 
cycle  surveys:  the  1978  DoD  Survey  of  Officers  and  Enlisted  Personnel, 
which  focused  on  the  in-service  population,  i.e.,  the  men  and  women  on 
active  duty  in  the  four  services.**  ana  the  1979  DoD  Survey  of 
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The  hang- •ccrporatiPir,- 


*See:  Zahava  D.  _ 

Research  Program,  Fiscal  Years  T5T8- 

K-i  i'65-mL,  Deceff.Trer Tyre-: - 

••See:  William  P.  hutzler  and  Zahava  D.  Doering,  197b  DoD  Survey 
of  Officers  and  Enlisted  Personnel;  Sample  Design  and  SelfeCtTgrr; 


The  haha  Lortoration,  n-iLbd-muM.,  February  1990;  zahava  b.  uoering, 
David  l..  Grissmer,  Jennifer  A.  hawes,  and  william  P.  Fiutzler,  1976 


iv 


Personnel  Entering  Military  Service,  which  was  administered  to  enlistees 
immediately  after  they  were  sworn  in.* 

The  1979  Reserve  Force  Studies  Surveys  were  aaministered  to 
enlisted  personnel  and  officers  in  the  Army  Reserve  and  Army  National 
Guard.  This  note  documents  the  procedures  used  to  design  and  administer 
the  surveys.  In  addition,  it  describes  the  fieldwork  and  assesses  the 
success  of  the  survey  administration  procedures  ana  the  appropriateness 
of  this  data  collection  model  lor  future  surveys  of  the  Reserve  Force 
population.  The  note  explains  design  and  implementation  of  the  1979 
Reserve  Force  Studies  Surveys  and  provides  a  basic  reference  document 
for  future  data  collection  from  Reserve  Force  personnel.  The  author  was 
responsible  for  the  technical  coordination  and  management  of  the  survey 
operations. 
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SUMMARY 


This  note  provides  documentation  for  the  1979  Reserve  Force  Studies 
Surveys ,  one  of  a  series  of  interrelated  data  collection  efforts  of  the 
Rand-DoD  Survey  Group,  a  component  of  Rand's  Manpower,  Mobilization,  and 
Readiness  Program.  The  1979  Reserve  Force  Studies  Surveys  were 
designed  to  be  the  first  of  a  series  of  periodic  surveys  administered  to 
the  Reserve  Force.  Prior  to  these  surveys,  the  Reserve  Force  was  not 
surveyed  regularly,  as  was  the  active  force.  This  note  documents  the 
procedures  used  to  design  and  administer  the  surveys  and  evaluates  the 
success  and  appropriateness  of  the  data  collection  model  for  future 
surveys  with  the  Reserve  Force  population.  It  will  be  useful  in  the 
design  and  administration  of  future  surveys  of  Reserve  Force  personnel. 

DESIGN  AND  ADMINISTRATION  OF  THE  SURVEY 

The  1979  Reserve  Force  Studies  Surveys  were  administered  to  a 
cross-section  of  enlisted  personnel  and  unit  commanders  in  both  the  Army 
Reserve  and  Army  National  Guard.  The  purpose  of  the  surveys  was  to 
collect  data  for  the  Office  of  the  Deputy  Assistant  Secretary  of  Defense 
(Reserve  Affairs) — ODASD(RA)  and  the  Army  Reserve  components  to  support 
policy  research  and  analysis  on  Reserve  Force  manning  problems. 

The  research  strategy  was  to  sample  441  Army  National  Guard  and  Army 
Reserve  units  of  different  authorized  strength,  in  different  geographical 
areas,  in  different  kinds  of  communities,  and  with  different  approaches 
to  solving  manning  problems.  We  collected  unit  personnel  data— from 
junior  and  senior  enlisted  members  and  unit  commanders— and  data 
describing  the  characteristics  of  the  unit  and  the  community. 

The  objectives  of  the  1979  Reserve  Force  Studies  Surveys  were  to: 

o  Collect  data  on  factors  operating  within  the  unit,  factors 
that,  when  combined  with  data  describing  the  surrounding 
community,  explain  differences  in  Reserve  Force  unit  manning 
levels. 
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o  Collect  data  on  factors  influencing  enlistment  and  reenlistment 
decisions. 

o  Collect  data  for  descriptive  statistics  from  a  representative 
sample  of  the  Reserve  Force  population. 

o  Provide  a  baseline  data  set  for  a  Reserve  Force  population 

sample  which  can  be  monitored  in  the  future  to  study  attrition. 


Four  questionnaires  were  designed  for  the  1979  Reserve  Force  Studies 
Surveys:* 

o  Form  1 :  1979  Reserve  Force  Personnel  Survey — FOR  JUNIOR 

ENLISTED  GRADES  E1-E4. 

o  Form  2:  1979  Reserve  Force  Personnel  Survey — FOR  SENIOR 
ENLISTED  GRADES  E5-E9. 

o  Form  3:  1979  Reserve  Force  Commander  Survey. 

o  Form  1979  Reserve  Force  Unit  Survey. 

Planning  for  this  study  began  in  January  1979.  The  surveys  were 
fielded  nationwide  in  late  November  1979  to  a  sample  of  224  Army  National 
Guard  units  and  217  Army  Reserve  units,  consisting  of  some  39,000 
enlisted  personnel  at  the  time  of  the  survey  administration.  Unit 
commanders  of  the  selected  units  were  responsible  for  data  collection 
during  regular  monthly  drill  assemblies.  Each  unit  was  responsible  for 
completing  one  Commander  Survey,  one  Unit  Survey,  and  Personnel 
Surveys  for  each  enlisted  member  of  the  unit. 

The  survey  administration,  originally  scheduled  to  be  completed  by 
December  1979,  was  delayed  until  July  1690  by  the  logistics  of  survey 
administration  during  drill  assemblies  and  operational  requirements,  such 
as  field  maneuvers,  and  emergency  call-ups.  After  the  survey,  question¬ 
naires  were  edited,  processed,  and  converted  into  machine-readable  files. 


*The  questionnaires  may  be  found  in  Zahava  D.  Doering,  David  W. 
Grissmer.  and  Jennifer  A.  Hawes,  1979  Reserve  Force  Studies  Surveys: 
Survey  Design,  Sample  Design,  and  AaffliniStratlTC '  Procedures',  The  Rand 
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CONCLUSIONS 

The  field  experience  with  the  1979  Reserve  Force  Studies  Surveys 
indicates  that  the  data  collection  procedures  developed  for  this  study 
worked  quite  well.  The  overall  response  rate  to  this  survey  was 
considerably  better  than  average  for  a  survey  of  Reserve  Force  personnel. 
We  attribute  this  success  to  the  effective  follow-up  of  sample  members  to 
obtain  the  highest  possible  response.  It  was  not  uncommon  for  previous 
surveys  of  Reserve  Force  personnel  to  achieve  response  rates  of  20  to  40 
percent.  Analysis  of  the  response  rates  to  the  1979  Reserve  Force 
Studies  Surveys  shows  that: 

o  In  both  the  Army  National  Guard  and  the  Army  Reserve,  the 
response  rates  for  the  Unit  and  Commander  Surveys  were 
quite  high,  averaging  80  percent. 

o  Sixty-five  percent  of  all  senior  enlisted  unit  members 
(Grades  E5-E9)  returned  completed  questionnaires. 

o  The  lowest  participation  rate  occurred  among  junior  enlisted 
unit  members  (Grades  E1-E4),  approximately  49  percent  of  whom 
returned  completed  questionnaires. 

o  The  response  rates  from  enlisted  unit  members,  while  not  as 
high  as  the  response  from  unit  commanders,  compared  favorably 
with  previous  data  collection  with  Reserve  Force  personnel 
and  with  similar  surveys  with  the  active  force  population. 

The  success  of  the  administrative  model  developed  for  the  1979 
Reserve  Force  Studies  Surveys  leads  us  to  conclude  that  surveys  with 
Reserve  Force  personnel  can  be  carried  out  successfully  if  the  required 
technical  and  administrative  support  is  available.  The  following 
elements  contributed  to  the  success  oi  this  data  collection  and  will  aid 
subsequent  surveys  of  Reserve  Force  personnel  and  other  military 
populations: 

o  Visible  support  from  top-level  military  officials. 

o  Use  of  military  channels  for  data  collection  and  sample 
accountability. 

o  Use  oi  military  time  for  survey  participation. 
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o  Well-defined  management  structure  for  the  survey  design  and 
administration. 

o  Extensive  questionnaire  pretesting, 

o  Detailed  administrative  instructions. 

o  Interactive  field  monitoring  with  follow-up  of  nonparticipants. 
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I.  INTRODUCTION 


The  1979  Reserve  Force  Studies  Surveys  make  up  one  of  several 
interrelated  data-collection  efforts  of  the  Rand-DoD  Survey  Group,  a 
component  of  the  Rana  Manpower,  Mobilization,  and  Readiness  Program, 
sponsored  by  the  Office  of  the  Assistant  Secretary  of  Defense  (Manpower, 
Reserve  Affairs  and  Logistics) — GASD(MRA&L).  The  survey  group's 
objectives  include  provision  of  policy-sensitive  information  about  the 
military  life  cycle.  The  military  life  cycle  includes  both  reserve  and 
active  force  enlistment  decisions,  career  orientations,  responses  to 
policies  that  affect  military  members  and  their  households,  and  decisions 
to  leave  the  military. 

To  date,  the  survey  group  has  designed  and  administered  three 
military  life  cycle  surveys: 

o  The  1979  DoD  Survey  of  Personnel  Entering  Military  Service, 
administered  to  enlistees  at  the  Armed  Forces  Entrance 
Examination  Stations  (AFEES)  immediately  after  the  enlistees 
were  sworn  in . 

o  The  1978  DoD  Survey  of  Officers  and  Enlisted  Personnel,  which 

focused  on  the  in-service  population,  i.e.,  the  men  and  women  on 
active  duty  in  the  four  services. 

o  The  1979  Reserve  Force  Studies  Surveys,  administered  to 

enlisted  personnel  and  officers  in  the  Army  Reserve  and  Army 
National  Guard . 

The  purpose  of  the  1979  Reserve  Force  Studies  Surveys  is  to 
provide  the  Office  of  the  Deputy  Assistant  Secretary  of  Defense  (Reserve 
Affairs) — ODASD(RA)  and  the  Army  Reserve  components  with  data  to  support 
policy  formulation  and  research  on  Reserve  Force  manning  problems.  The 
data  gathered  in  the  1979  Reserve  Force  Studies  Surveys  will  permit 
analyses  of  the  factors  underlying  the  success  or  failure  of  unit  manning 
and  readiness  and  will  provide  a  unique  description  of  the  men  and  women 
currently  in  the  Army  heserve  components. 
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Our  research  strategy  was  to  sample  441  Army  National  Guard  ana  Army 
Reserve  units  of  different  authorized  strength  levels,  in  different 
geographical  areas,  in  different  kinds  of  communities,  and  with  different 
approaches  to  solving  manning  problems.  We  collected  unit  personnel 
data — from  junior  and  senior  enlisted  members  and  unit  commanders — as 
well  as  data  describing  the  characteristics  of  the  unit  and  the 
community.  Personnel  data  will  illuminate  the  factors  affecting  the 
decision  to  enlist  and  reenlist  in  the  Reserve  Force.  These  factors 
include  family  income,  past  military  history,  and  potential  conflicts 
between  civilian  job  requirements  and  reserve  participation.  We 
collected  data  describing  reserve  unit  characteristics  that  encourage  and 
discourage  the  enlistment  and  reenlistment  of  local  residents,  including 
recruiting  resources  and  strategies,  equipment  and  facilities  available 
for  training,  unit  commander  attitudes,  and  available  personnel 
management  initiatives.  Community  data  include  population,  types  and 
wages  of  available  jobs,  and  the  existence  of  other  Reserve  Force  units. 

The  specific  objectives  of  the  1979  Reserve  Force  Studies  Surveys 
were  to: 

o  Collect  data  on  factors  operating  within  the  unit,  factors  that, 
when  combined  with  data  describing  the  surrounding  community, 
explain  differences  in  Reserve  Force  unit  manning  levels. 

o  Collect  data  on  factors  influencing  enlistment  and  reenlistment 
decisions. 

o  Collect  data  for  descriptive  statistics  from  a  representative 
sample  of  the  Reserve  Force  population. 

o  Provide  a  baseline  data  set  for  a  Reserve  Force  population 

sample  which  can  be  monitored  in  the  future  to  study  attrition. 

Because  of  the  multiple  purposes  of  these  surveys  and  the  large 
number  of  questions  required  to  address  some  of  the  topics,  four  separate 
questionnaires  were  designed  for  the  1979  Reserve  Force  Studies 
Surveys : 


Form  1 ,  1979  Reserve  Force  Personnel  Survey — FOR  ENLISTED 
GRADES  L1-E4,  collected  data  from  all  junior  enlisted 


3 


personnel  who  were  members  of  the  sample  units  at  the  time  of 
the  survey  administration.  The  major  part  of  the  information 
collected  in  this  questionnaire  related  to  the  first-term 
enlistment  decision  and  to  the  characteristics  and  experiences 
of  the  individual  prior  to  enlistment.  A  group  of  possible 
indicators  of  attrition  and  reenlistment  behavior,  together  with 
detailed  economic  and  civilian  labor  force  data,  were  also 
included . 

o  Form  2,  1979  Reserve  Force  Personnel  Survey — FOR  ENLISTED 
GRADES  E5-E9,  was  administered  to  all  senior  enlisted 
personnel,  including  unit  technicians  and  training  NCOs 
(noncommissioned  officers),  who  were  members  of  the  sampled 
units  at  the  time  of  the  survey  administration.  This  question¬ 
naire,  like  Form  1,  also  collected  detailed  economic  and 
civilian  labor  force  data  and  possible  indicators  of  reenlist¬ 
ment  behavior.  In  addition,  a  major  portion  of  the  question¬ 
naire  focused  on  the  individual's  past  military  experiences — 
both  active  and  reserve — and  his  perceptions  of  the  unit  envi* 
ronment  and  manning  problems. 

o  Form  3.  1979  Reserve  Force  Commander  Survey,  was  completed  by 
all  unit  commanders  of  the  sampled  units.  This  survey 
collected  information  about  the  commanders'  characteristics, 
military  and  civilian  backgrounds,  and  opinions  about  unit 
activities  and  environment. 

o  Form  4,  1979  Reserve  Force  Unit  Survey,  covered  basic  factual 
information  about  each  sampled  unit.  This  information  was 
provided  by  either  the  unit  commander  or  another  unit  member  who 
was  familiar  with  unit  data,  usually  the  unit  technician  or  unit 
clerk. 

Planning  for  the  surveys  began  in  January  1979.  The  surveys  were 
fielded  nationwide  in  late  November  1979  to  the  sampled  224  Army  National 
Guard  units  and  217  Army  Reserve  units,  consisting  at  the  time  of  the 
survey  of  approximately  39,000  enlisted  personnel.  Unit  commanders  of 
the  selected  units  were  responsible  for  collecting  data  during  regular 
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monthly  drill  assemblies.  Each  unit  commander  was  responsible  for 
completing  one  Commander  Survey,  one  Unit  Survey  and  administering 
Personnel  Surveys  to  all  enlisted  unit  members  (Grades  E1-E9).  The 
surveys  were  scheduled  for  completion  in  December  1979.  The  logistics  of 
survey  administration  during  drill  assemblies  and  such  operational 
requirements  as  unit  maneuvers  and  emergency  call-ups  delayed  completion 
of  the  data  collection  until  July  1980.  After  the  survey,  questionnaires 
were  edited,  processed,  and  converted  into  machine-readable  files. 

The  documentation  for  this  survey  is  contained  in  three  Rand  notes. 
The  first  describes  the  survey  and  sample  design  and  administrative 
procedures.*  The  second  note,  the  user's  manual  and  codebook  for  all 
four  questionnaires,**  summarizes  the  sample  design,  provides  preliminary 
information  about  response  rates  and  general  information  about  the  survey 
questionnaires  and  their  processing,  and  describes  the  data  in  detail. 

In  this  second  note,  users  will  find  the  information  needed  to  interpret 
and  analyze  coded  responses  to  each  question,  as  well  as  technical 
information  about  the  structure  of  the  data  files,  including  the  number 
of  records. 

This  is  the  third  note  in  the  series  of  documentation  for  the  1979 
Reserve  Force  Studies  Surveys.  Section  II  of  this  note  summarizes  the 
major  survey-related  activities  from  the  start  of  the  project  to  the 
final  preparation  of  the  analytic  data  files  resulting  from  the  data 
collection,  the  tasks  involved  in  the  design  and  execution  of  the  reserve 
surveys,  and  the  technical  and  administrative  support  needed  for  the 
survey.  Section  III  describes  the  model  developed  for  monitoring  the 
survey  administration  and  the  operational  organization  for  those 
activities,  the  issues  and  events  between  the  initial  mailing  of  survey 
materials  in  November  1979  and  the  receipt  o t  the  last  completed  ques¬ 
tionnaire  in  July  I960,  the  problems  encountered  in  the  distribution  and 
return  ol  surveys,  and  the  follow-up  strategies  used  during  the  seven- 
month  fieldwork  period  to  increase  survey  response  rates.  Section  IV 


•Zahava  D.  Doering,  David  k.  Grissmer,  and  Jennifer  A.  Hawes,  1979 
Reserve  Force  Studies  Surveys:  Survey  Design,  Sample  Design  and 
Administrative  procedure's,  Thy  Band  corporation,  n-17U9-ctal,  August 
T78H - 

••Zahava  D.  Doering,  David  k.  Grissmer,  and  Jennifer  A.  Hawes,  1979 
Reserve  Force  Studies  Surveys:  User's  hanual  and  Codebook,  The  Rand 
Cbrpurs t lOh ,  N— 1755-MKAL,  September  1981. 
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assesses  the  success  of  the  fieldwork  procedures,  and  Section  V  general¬ 
izes  the  lessons  learned  from  the  survey  and  their  implications  for 
subsequent  data  collection  efforts  involving  the  Reserve  Force  and  other 
military  populations. 

Appendix  A  contains  a  complete  description  of  all  survey-related 
activities  undertaken  to  design  and  administer  the  1 975  Reserve  Force 
Studies  Surveys,  including  the  time  required  to  complete  each  task. 
Appendix  B  contains  the  follow-up  letters  and  notices  sent  by  Rand  and 
the  Army  Reserve  components  to  encourage  survey  participation  by  the 
selected  units. 


II.  TASKS  FOR  SURVEY  DESIGN  AND  IMPLEMENTATION 


BACKGROUND 

The  design  and  implementation  of  the  1979  Reserve  Force  Studies 
Surv e> s  involved  eight  major  tasks:  (1)  preliminary  survey  planning, 
from  which  the  basic  operational  plan  for  conducting  the  surveys  evolved; 
(2)  questionnaire  preparation  and  pretesting;  (3)  sample  design  and 
implementation;  (M)  survey  administrative  design;  (5)  data  collection  and 
monitoring;  (6)  receipt  and  editing  of  completed  questionnaires;  (7)  data 
processing;  and  (8)  documentation  of  survey  procedures.  These  tasks 
entailed  numerous  interrelated,  time-consuming  activities,  each  ol  which 
required  careful  planning  and  control  by  staff  at  Rand,  the  Department  of 
Defense,  and  the  Army  Reserve  components  responsible  for  various  aspects 
of  the  survey  effort. 

The  various  survey  activities  can  be  divided  into  two  groups:  (1) 
those  that  are  sequential  and  depend  on  the  completion  of  a  prior  step 
(e.g.,  data  collection  cannot  start  until  after  the  questionnaire  and 
sample  plan  have  been  finalized)  and  (2)  those  that  can  be  carried  out 
more  or  less  independently  of  other  steps,  but  depend  on  staff  and 
resources  needed  for  conducting  other  survey  tasks  (e.g.,  the  final 
implementation  of  the  sampling  plan  was  conducted  at  the  same  time  the 
questionnaires  were  being  revised  and  put  into  final  form).  Understand¬ 
ing  this  dual  classification  of  activities  into  "dependent"  and 
"independent"  survey  processes  was  critical  in  planning  the  survey, 
because  the  resource  and  time  requirements  had  to  be  identified  and 
provided  for  at  the  start  of  the  project. 

The  remainder  of  this  section  describes  in  detail  the  tasks  required 
to  carry  out  the  1979  Reserve  Force  Studies  Surveys  and  identifies 
issues  that  should  be  taken  into  account  in  planning  subsequent  surveys  of 
Reserve  Force  personnel.  Appendix  A  contains  a  complete  description  of 
all  survey-related  activities  undertaken  to  design  and  implement  the 
1979  Reserve  Force  Studies  Surveys. 
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TASK  1:  PRELIMINARY  SURVEY  PLANNING 

Planning  for  this  survey  began  in  January  1979.  After  the  information 
needs  for  the  study  had  been  identified  and  a  determination  made  that 
existing  data  were  inadequate,  the  first  step  in  planning  for  the  survey 
was  to  layout  the  specific  research  objectives  of  the  investigation. 

Members  of  Hand's  Design  and  Analysis  Group  prepared  an  internal 
discussion  paper  on  the  policy  and  research  requirements  for  the  surveys 
and  proposed  a  design  for  the  data  collection.  The  exact  methodology  for 
carrying  out  the  data  collection  was  then  proposed  and  developed  by 
members  of  the  Hand-DoD  Survey  Group.  The  survey  staff  produced  an 
internal  survey  planning  document  which  (1)  identified  the  technical  and 
administrative  support  required  for  the  successful  implementation  of  the 
surveys,  (2)  described  the  steps  involved  in  carrying  out  the  survey,  (3) 
proposed  a  schedule  for  completing  the  survey  work,  and  finally,  (4) 
delineated  staff  responsibility  for  various  aspects  oi  the  survey  work, 
with  emphasis  on  the  division  of  responsibility  between  staff  at  Rand 
(both  analysts  and  survey  professionals),  the  Department  of  Defense 
(DoD),  and  the  participating  Army  Reserve  components. 

The  two  internal  planning  documents  produced  at  the  start  of  this 
research  project  proved  to  be  crucial  in  the  design  and  implementation 
of  the  survey  for  several  important  reasons.  First,  the  planning 
materials  formed  the  preliminary  operational  plans  for  conducting  the 
entire  survey.  The  documents  established  the  basic  planning  assumptions 
for  the  overall  survey  and  sample  design,  content  of  the  questionnaires, 
the  data  collection  procedures,  and  the  data  processing  requirements,  as 
well  as  information  about  required  staff  and  resource  levels  needed  to 
field  the  survey  in  a  timely  fashion.  From  these  preliminary  opera¬ 
tional  plans,  the  final  research  strategy  for  conducting  the  survey 
evolved.  The  planning  documents  also  served  as  an  initial  outline  of 
the  final  survey  and  analytic  reports  which  were  scheduled  for  comple¬ 
tion  at  the  end  of  the  study. 

Second,  these  documents  served  as  a  basis  for  initiating  a  dialogue 
with  Rand  analysts  and  the  client  on  the  proposed  survey  plans.  Planning 
meetings  were  helo  with  participating  staff  at  Rand,  DoD,  and  the  Army 
Reserve  components  to  discuss  the  content  of  the  planning  reports  snd  to 
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clearly  define  the  administrative  and  technical  support  needed  from  each 
organization.  Obtaining  early  agreement  on  the  proposed  operational 
plans  and  commitments  to  fully  support  the  survey  effort  was,  of  course, 
one  of  the  most  important  features  of  the  survey  planning  process. 

A  third  function  of  the  internal  planning  documents  was  to  help 
staff  participating  in  the  survey  effort  to  better  understand  the 
procedures  necessary  to  field  the  survey  and  the  time  needed  for  each 
stage  of  the  survey. 

The  last  function  served  by  the  internal  planning  documents  was  to 
establish  the  management  structure  for  the  survey  work.  Maximizing  the 
analytic  value  of  the  collected  data  required  close  coordination  between 
the  Rand  Design  and  Analysis  Group,  which  was  responsible  for  the  analytic 
design  of  the  surveys,  the  Rand-DoD  Survey  Group,  which  had  responsibility 
for  the  overall  survey  coordination,  and  various  staff  at  DoD  and  the 
Army  Reserve  components  who  provided  much  needed  technical  and  adminis¬ 
trative  support  for  the  study. 

TASK  2:  QUESTIONNAIRE  PREPARATION 

Although  the  development  of  questionnaires  for  this  study  can  be 
described  quite  simply,  the  process  involved  many  time-consuming  steps 
and  many  technical  issues  had  to  be  addressed. 

Preparatory  work  for  the  questionnaire  design  began  during  the 
preliminary  survey  planning  phase.  As  described  above,  the  preliminary 
data  requirements  for  the  questionnaires  were  identified  early  in  the 
planning  process.  At  that  time,  teams  of  Rand  analysts  and  survey 
professionals  conducted  informal,  semistructured  interviews  with  Reserve 
Force  personnel  to  further  refine  the  data  requirements  for  the  study  so 
that  draft  questionnaires  could  be  prepared. 

While  the  analysts  reviewed  and  modified  the  data  requirements  for 
the  study  on  the  basis  of  the  preliminary  interviews  with  Reserve  Force 
personnel,  the  survey  staff  reviewed  past  data  collection  methods  and 
formats  for  collecting  such  data.  This  review  included  a  close 
examination  of  previous  data  collection  from  Reserve  Force  personnel,  as 
well  as  comparable  surveys  of  the  active  force  population  and  related 
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civilian  survey  efforts,  and  proved  useful  in  the  survey  design  process. 
The  Survey  Group  then  prepared  draft  questionnaires  based  on  revised 
design  specifications  provided  by  the  analysts.  Rand  analysts 
knowledgeable  about  Reserve  force  policy  issues,  staff  members  from  the 
Office  of  the  Deputy  Assistant  Secretary  of  Defense  (Reserve  Affairs), 
and  Army  Reserve  components  personnel  who  had  participated  in  the  data 
specifications  reviewed  the  draft  questionnaires. 

After  draft  questionnaire  items  were  thoroughly  reviewed  at  Rand 
and  DoD,  they  were  revised  and  prepared  for  formal  pretesting  with 
representative  samples  of  the  Reserve  force  population.  A  large  scale 
pretest  was  conducted  with  about  80  Army  Reserve  component  members  to 
identify  any  technical  and  substantive  problems  with  the  questionnaire 
items.  Immediately  upon  completion  of  the  pretesting,  the  survey  staff 
documented  the  pretest  experience  with  emphasis  on  the  implications  for 
further  questionnaire  modifications.  The  data  collection  instruments 
were  again  revised  and  short  pretests  were  conducted  to  ensure  that 
problematic  concepts,  wording,  and  formats  had  been  corrected. 

The  questionnaires  were  then  put  into  final  form  in  preparation  for 
printing.  At  this  stage  we  considered  a  number  of  design  factors  to 
minimize  the  respondent  burden  in  completing  the  questionnaire  and  to 
facilitate  handling  of  the  questionnaire  both  in  the  field  and  in  data 
processing.  The  questionnaires  were  structured  to  minimize  the  need  for 
complicated  skip  patterns  and  intricate  questionnaire  designs.  Question 
formats  and  the  layout  of  the  entire  survey  instrument  were  designed  to 
be  as  simple  as  possible  to  administer,  consistent  with  the  data  and 
optical  scanning  processing  requirements.  Unduly  long  and  complex  self- 
administered  questionnaires  are  apt  to  make  respondents  reluctant  to 
participate  in  a  survey.  We  therefore  designed  the  final  question¬ 
naires  to  be  as  easy  as  possible  for  respondents  to  fill  out,  so  as  to 
minimize  refusals  and  incomplete  questionnaires. 

Before  the  final  questionnaires  could  be  given  to  the  contractor 
responsible  for  printing  and  distribution,  the  survey  staff  had  to  obtain 
a  Reports  Control  Symbol  (RCS)  through  the  Office  of  the  Deputy  Assistant 
Secretary  of  Defense  (Reserve  Affairs)  from  the  Department  of  Defense 
(DoD)  Comptroller's  office.  This  RCS  number  served  as  official  DoD 
approval  for  the  survey.  Our  RCS  request  was  submitted  immediately  after 
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the  pretest,  when  the  questionnaires  and  operational  plans  for  the  study 
were  in  near-final  form.  The  RCS  submission  contained  the  following 
information:  (1)  a  justification  statement  explaining  the  purpose  of  the 
survey,  data  collection  plans,  individual  and  command  time  required,  and 
a  schedule  of  events  and  (2)  the  most  recent  draft  of  the  questionnaires 
»  and  administrative  instructions  to  be  used.  Within  three  weeks  we 

received  an  RCS  number,  which  was  then  aoded  to  the  questionnaires  and  the 
procedures  given  to  service  administrators.  The  RCS  number  was  also  used 
on  correspondence  between  Rand,  the  sampled  units,  and  their 
organizational  superiors. 

Once  we  had  received  a  survey  clearance  number,  the  final 
i  questionnaires  were  sent  to  the  printing  contractor,  together  with 

technical  specifications  for  the  question  order,  question  formats,  and 
color  selections  for  different  forms.  All  printer-ready  data  collection 
instruments  were  reviewed  and  approved  by  the  survey  staff  prior  to  the 
final  printing.  The  printing  contractor  was  also  responsible  for 
distributing  the  questionnaires  and  accompanying  materials  needed  for 
survey  administration  directly  to  each  sampled  unit. 

TASK  3:  SAMPLE  DESIGN  AND  IMPLEMENTATION 

The  sample  design  and  implementation  stage  included  all  of  the 
activities  required  to  design  a  sampling  plan  and  select  a  sample  which 
would  meet  the  analytic  requirements  of  the  data  collection.  A  great 
deal  of  coordination  was  required  between  the  Rand  Design  and  Analysis 
team,  the  Rand  survey  staff,  and  personnel  at  DoD  and  the  Army  Reserve 
components  responsible  for  various  aspects  of  the  sample  work.  The 
sample  plan  was  developed  by  the  Rand  Design  and  Analysis  Group  and 
procedures  for  selecting  the  sample  by  the  Rand-DoD  Survey  Group.  Survey 
support  functions,  including  programming,  preparation  of  sample  files, 
and  procurement  of  survey  administrative  materials,  were  performed  by  the 
Defense  Manpower  Data  Center  (DMDC) — the  DoD  agency  which  has  the 
authority  to  maintain  and  use  military  personnel  records.  Technical 
support  for  resolving  sample  problems  was  provided  by  staff  of  the  Army 
Reserve  components.  The  major  tasks  involved  in  the  sample  design  and 
implementation  are  discussed  below. 
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A  preliminary  sample  design  and  implementation  plan  was  developed 
early  in  the  survey  planning  process.  Basic  decisions  about  the  overall 
sampling  strategy  had  to  be  made  before  the  data  requirements  for  the 
questionnaires  and  survey  administration  plans  could  be  finalized.  For 
this  reason,  substantive  reviews  of  the  proposed  sampling  plan  were 
conducted  at  the  start  of  the  project  and  early  agreement  was  reached  on 
the  basic  sampling  approach. 

As  soon  as  work  began  on  the  sample  design,  Rand  requested  and 
received  primary  points-of-contact  in  the  Army  Reserve  components  and  at 
DM DC  to  assist  Rand  in  the  sample  implementation.  Planning  meetings 
were  held  with  the  sample  contacts  to  discuss  the  overall  sampling  and 
implementation  strategy,  to  identify  the  technical  and  administrative 
support  needed  from  the  contacts,  and  to  review  planning  schedules  for 
completing  major  sampling  tasks.  Rand  also  provided  DMDC  with  an 
internal  planning  document  that  (1)  outlined  all  of  the  sample  data 
processing  activities  to  be  performed  by  DMDC  staff  and  provided  a 
schedule  to  assist  them  in  planning  resources  for  the  sampling  activi¬ 
ties  and  (2)  served  as  a  basis  for  initiating  a  dialogue  with  our  DMDC 
liaison,  who  provided  useful  comments  and  suggested  revisions  in  the 
implementation  strategy  which  improved  the  procedures. 

Following  substantive  reviews,  the  Rand  Design  and  Analysis  Group 
revised  the  preliminary  sample  plan  on  the  basis  of  suggestions  from 
participating  staff  and  results  from  the  questionnaire  pretests.  When 
the  sample  design  was  finalized,  the  sample  of  units  was  selected.  The 
survey  staff  developed  procedures  and  materials  for  all  sample  implemen¬ 
tation  activities  and  coordinated  sample-related  requirements  with  the 
Army  Reserve  components  and  DMDC. 

Upon  receipt  of  the  Rand  sampling  list,  DMDC  staff  verified  unit 
mailing  addresses,  prepared  sample  lists  and  rosters  containing  names  of 
unit  personnel,  and  produced  other  materials  needed  for  the  survey 
administration.  Staff  at  the  Army  Reserve  components  assisted  in  the 
resolution  of  sample  problems,  such  as  identifying  inactivated  units  and 
solving  address  problems.  Upon  completion  of  all  implementation 
activities,  DMDC  sent  sample  tapes  and  sample  accountability  forms  to 
the  mailing  and  processing  contractor. 
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TASK  M:  SURVEY  ADMINISTRATIVE  DESIGN 

The  survey  administrative  design  phase  included  all  of  the  steps 
involved  in  setting  up  a  model  and  procedures  for  the  data  collection. 

The  technical  coordination  and  design  of  administrative  procedures  was 
the  responsibility  of  the  Rand-DoD  Survey  Uroup.  The  actual  data 
collection  was  the  responsibility  of  the  commanding  officer  of  each 
sample  unit.  An  administrative  model,  which  had  top-level  military 
support,  was  needed  to  ensure  that  the  execution  of  the  survey  in  the 
field  corresponded  to  the  design  requirements.  Obviously,  poor  execu¬ 
tion  of  the  survey  in  the  field  could  have  seriously  damaged  the  survey 
results.  Past  surveys  of  Reserve  Force  personnel  have  encountered  serious 
problems  in  obtaining  administrative  and  respondent  support  because 
routine  channels  for  survey  coordination  do  not  exist  in  the  Army  Reserve 
components.  Recognizing  these  problems,  we  developed  an  overall  approach 
to  gaining  support  for  the  survey  effort.  The  remainder  of  this  section 
discusses  the  design  of  administrative  procedures  for  the  survey. 

The  operational  requirements  for  survey  administration  were  defined 
by  members  of  the  Rand  Design  and  Analysis  team  as  part  of  the  initial 
survey  planning.  The  survey  group  then  examined  alternative  methodol- 
gies  for  data  collection  and  reviewed  past  efforts  to  collect  data  froia 
reserve  and  active  force  populations.  Based  on  this  review,  the  survey 
staff  developed  a  preliminary  administrative  plan. 

The  first  step  in  setting  up  an  administrative  model  for  the  data 
collection  was  to  ask  our  client,  the  Office  of  the  Deputy  Assistant 
Secretary  of  Defense  (Reserve  Affairs) ,  to  establish  an  interface  with 
the  Army  Reserve  components  through  identified  primary  points-of-contact 
(PPOC).  In  March  1979,  a  planning  meeting  was  held  with  the  PPOCs  to 
discuss  the  general  administrative  requirements  and  to  solicit  their 
support  of  the  survey  effort.  We  requested  and  received  support  from  the 
PPOCs  in  three  basic  areas: 

o  DoD  and  the  Army  Reserve  components  would  provide  top-level 
support  of  military  officials  by  (1)  endorsing  the  survey  by 
sending  notification  letters  to  sampled  units  and  their  organi- 
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zational  superiors  and  (2)  allocating  staff  and  resources  to 
assist  Hand  in  the  survey  implementation  (e.g.,  arranging  for 
pretests,  resolving  sampling  problems,  and  assisting  in  monitor-* 
ing  the  field  administration) . 

o  DoD  and  the  Army  Reserve  components  would  commit  administrative 
resources  for  carrying  out  the  actual  data  collection.  To 
improve  response  rates  and  sample  accountability,  units  would  be 
held  responsible  for  the  data  collection.  The  commanding 
officer  of  each  sampled  unit  would  be  the  person  responsible 
for  the  actual  data  collection. 

o  The  information  to  be  collected  from  keserve  Force  personnel 
was  of  sufficient  importance  to  DoD  and  the  Army  Reserve 
components  so  that  duty  time  (regular  monthly  drill 
assemblies)  was  to  be  used  for  group  or  individual 
administration. 

In  consultation  with  the  PPOCs,  the  survey  group  developed  written 
procedures  and  administrative  materials.  To  minimize  the  administrative 
burden  on  sample  units,  Rand  developed  simple  survey  accountability 
procedures  and  forms.  Staff  at  the  Defense  Manpower  Data  Center  provide.5 
the  materials  needed  at  the  unit  level  for  the  administration  and 
of  the  surveys. 

TASK  5:  DATA  COLLECTION  AND  MONITORING 

From  December  1979  through  June  I960,  the  survey  staff  monitored  the 
implementation  of  the  survey  in  the  field,  establishing  quality  control 
measures  in  consultation  with  our  service  PPOCs.  First,  Rand  established 
contact  with  sampled  units  in  advance  of  the  actual  survey  to  verify  that 
each  unit  had  been  informed  of  the  survey  through  military  channels. 

Rand  then  sent  advance  letters  to  each  sample  unit  to  request  a  unit 
point-of-contact  (and  telephone  numbers)  and  to  verify  the  number  of 
questionnaires  needed  for  the  survey  administration.  Once  questionnaires 
and  accompanying  survey  materials  had  been  distributed.  Hand  set  up 
monitoring  procedures  to  verify  that  each  unit  had  received  the  survey 
shipment.  In  a  few  instances,  supplemental  survey  shipments  were  sent  to 
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replace  lost  or  misplaced  questionnaires.  As  the  sampled  units  returned 
the  completed  surveys,  Rand  assessed  preliminary  survey  response  rates 
and  identifed  potential  fieldwork  problems,  such  as  incomplete  or  partial 
shipments  and  nonparticipating  units. 

To  maximize  survey  response  rates,  we  developed  plans  in  conjunction 
with  our  service  PPOCs  to  follow  up  on  units  that  returned  incomplete  or 
partial  shipments  or  failed  to  return  any  surveys.  We  found  it  necessary 
to  contact  all  nonparticipating  units  by  telephone  and  follow-up 
letters  and  notices  (e.g.,  military  messages  sent  by  the  Army  Reserve 
components  and  Rand  follow-up  letters)  in  order  to  obtain  completed 
questionnaires  from  units  that  were  inclined  to  refuse.  Our  field 
monitoring  procedures  were  designed  to  identify  fieldwork  problems  early 
so  that  appropriate  strategies  for  maximizing  survey  participation  could 
be  developed  as  quickly  as  possible.  To  encourage  participation,  we 
extended  the  fieldwork  period  several  times  to  allow  units  greater 
flexibility  in  scheduling  the  survey  administration.  Throughout  the 
field  period,  our  military  PPOCs  maintained  regular  contact  with  the  Rand 
survey  staff  and  assisted  the  field  monitoring  by  contacting  all  problem 
units  either  by  telephone  or  follow-up  letters  requesting  cooperation. 
(Appendix  B  contains  copies  of  the  follow-up  letters  sent  to  encourage 
survey  participation.) 

Our  unit-level  monitoring  resulted  in  a  field  report  on  each 
nonparticipating  or  problem  unit  to  assist  us  in  identifying  solutions  to 
the  field  problems.  Throughout  the  field  period,  Rand  produced  regular 
survey  progress  reports  for  the  client  as  an  aid  in  monitoring  survey 
response  rates. 

TASK  6:  SURVEY  RETURNS  AND  EDITING 

As  units  completed  the  survey  administration,  they  returned 
completed  surveys  and  accompanying  materials  directly  to  the  Defense 
Manpower  Data  Center.  From  December  1979  through  June  I960,  DMDC 
received  well  over  20,000  completed  surveys,  which  staff  members  reviewed 
manually  and  prepared  for  data  processing.  The  procedures  for  handling 
the  completed  questionnaires  are  outlined  below. 

After  being  inventoried  and  logged  in  at  DMDC,  incoming  question¬ 
naires  were  given  a  Rand  unit  identification  number.  A  twofold  field 
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procedure  had  been  developed  for  the  survey  to  ensure  that  each  returned 
questionnaire  could  be  linked  to  a  sample  unit.  First,  all  units  were 
instructed  to  return  completed  questionnaires  in  a  batch,  together  with 
the  unit's  roster,  which  contained  the  unit's  military  designation  and 
mailing  address.  Second,  each  respondent  was  instructed  to  record  his  or 
her  unit  designation  and  location  on  the  questionnaire.  Thus,  if  the 
unit  failed  to  return  all  questionnaires  with  the  roster,  DMDC  staff 
could  check  the  questionnaire  to  ascertain  the  unit  identification.  The 
procedure  worked  exceptionally  well:  DMDC  was  able  to  assign  a  sampling 
unit  identifier  to  99  percent  of  all  returned  questionnaires. 

Enlisted  Personnel  Survey  (Forms  1  and  2)  were  also  carefully 
reviewed  for  suitability  for  processing  by  optical  scanning.*  This 
review  included  making  sure  that  the  questionnaires  did  not  contain  stray 
markings,  that  pencil  markings  were  dark  enough  to  be  optically  scanned, 
and  that  questionnaires  which  had  been  marked  with  ink  or  which  contained 
torn  or  missing  pages  were  identified.  Inked  questionnaires  and  other 
potentially  unscannable  questionnaires  were  re-marked  with  scannable 
pencils. 

After  editing,  Forms  1  and  2  were  sent  to  the  optical  scanning 
contractor,  Intran  Corporation,  for  the  preparation  of  machine-readable 
data.  To  simplify  processing  at  Intran,  DMDC  sorted  questionnaires  by 
form  number  prior  to  shipping. 

The  Unit  and  Commander  questionnaires  were  also  inventoried  and 
logged  in  at  DMDC;  they  were  then  sent  in  batches  to  Rand  to  be  edited 
and  prepared  for  subsequent  processing  by  keypunching.  The  objectives  of 
the  questionnaire  edit  were  to  ( 1 )  prepare  questionnaires  for  data  entry, 
(2)  assign  numeric  codes  to  respondents'  handwritten  entries,  (3)  review 


•The  two  questionnaires  for  enlisted  unit  members,  Forms  1  and  2, 
were  self-administered  forms  suitable  for  subsequent  processing  by 
optical  scanning.  Most  of  the  questions  were  precodea;  that  is,  they 
required  the  respondent  to  select  the  appropriate  answer  from  a  set  of 
printed  responses  and  to  darken  the  "bubble"  on  the  questionnaire 
corresponding  to  that  answer.  Others,  known  as  grid  questions,  required 
the  respondent  to  enter  a  multidigit  number  in  a  set  of  boxes  by 
darkening  bubbles  corresponding  to  each  appropriate  digit.  The  optical 
scanner  reads  every  mark  (darkened  bubble)  made  by  the  respondent  on  the 
questionnaire.  Numeric  values  are  then  assigned  to  each  mark  read  by  the 
optical  scanner  to  transform  questionnaire  data  into  computer  tape  data 
for  analysis. 
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marginal  comments  provided  by  the  respondents,  and  (4)  identify  any 
questions  in  which  the  respondent  marked  more  than  one  answer,  but  for 
which  only  one  answer  was  allowed.  All  such  problems  had  to  be  resolved 
by  survey  and  analytic  staff  prior  to  data  entry.  Upon  completion  of 
editing,  the  Unit  and  Commander  questionnaires  were  returned  to  DMDC  for 
keypunching  and  preparation  of  machine-readable  data. 

TASK  7:  DATA  PROCESSING 

In  March  1979,  as  soon  as  the  preliminary  data  requirements  for  the 
surveys  were  identified,  a  final  decision  was  made  about  the  most  cost 
efficient  processing  mode  for  the  surveys.  Sample  sizes,  budget 
constraints,  and  the  general  suitability  of  the  data  requirements  to 
alternative  data  collection  modes  were  considered.  We  determined  that 
two  different  data  processing  modes  were  appropriate  for  these  surveys. 
The  Enlisted  Personnel  Surveys,  Forms  1  and  2,  were  designed  for  optical 
scanning;  the  Unit  and  Commander  Surveys,  Forms  3  and  4,  were  designed 
for  keypunching. 

Well  in  advance  of  the  data  collection,  DMDC  procured  an  optical 
scan  contractor  to  handle  the  distribution  and  initial  processing  for  the 
Enlisted  Personnel  surveys.  Under  a  special  contractural  arrangement, 
DMDC  selected  an  optical  scan  contractor  who  provided  data  processing 
support  for  the  present  reserve  study  as  well  as  all  other  surveys  under 
DMDC's  jurisdiction  during  that  fiscal  year.  Rand  provided  DMDC  with  the 
technical  specifications  for  the  work  to  be  performed  by  the  contractor. 
This  included  detailed  information  pertaining  to  clerical  tasks  involved 
in  the  distribution  and  return  of  questionnaires  together  with  design 
specifications  for  the  size,  length,  and  printing  quantities  for  the 
survey  instruments.  DMDC  staff  keypunched  all  of  the  data  for  the  unit 
and  commander  surveys  and  produced  the  final,  edited  analytic  data  files 
for  all  four  variants  of  the  questionnaire. 

The  Rand-DoD  Survey  Group  provided  all  technical  specifications  for 
the  data  entry,  data  "cleaning,"  and  final  preparation  of  machine- 
readable  survey  files  and  coordinated  all  data  processing  activities. 

As  completed  questionnaires  were  returned  by  sample  units,  they  were 
initially  processed  either  by  keypunching  or  optical  scanning  and  a 
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preliminary  computer  file  created.  Additional  computer  editing  at  DMDC 
was  carried  out  to  handle  missing  and  unanswered  items  and  to  alter 
inconsistent,  or  impossible  entries,  e.g.,  the  age  of  six  entered  for  a 
unit  commander.  When  a  "clean"  file  was  finally  produced,  the  survey 
data  was  in  analyzable  form. 

TASK  8:  SURVEY  DOCUMENTATION 

In  designing  and  implementing  the  1979  Reserve  Force  Studies 
Surveys ,  the  survey  staff  made  a  conscious  effort  to  ensure  that 
information  and  data  would  be  available  to  describe  and  evaluate  various 
aspects  of  the  survey  process.  The  methodology  used  in  military  data 
collection  activities  can  be  improved  through  the  wide  dissemination  of 
methodological  studies,  such  as  the  ones  produced  for  this  study.  The 
experience  gained  in  this  survey  has  definite  implications  for  subsequent 
survey  efforts  in  the  Army  Reserve  components,  as  well  as  future  surveys 
of  other  military  personnel. 

CONCLUSIONS 

As  we  have  seen,  conducting  the  1979  Reserve  Force  Studies 
Surveys  required  many  activities,  each  of  which  had  to  be  carefully 
planned,  coordinated  and  monitored  throughout  the  study.  The  major 
factors  that  should  be  taken  into  account  in  planning  staff  time  and 
resources  for  similar  surveys  is  summarized  below.  These  factors  pertain 
to  the  survey-related  aspects  of  the  study,  rather  than  to  the  allocation 
of  resources  for  initial  concept  development  and  data  analysis. 

1.  Survey  staff  time  for  planning  the  study  and  guiding  it  through 
the  various  stages  described  in  this  document. 

2.  Labor  and  material  costs  for  pretesting  ana  evaluating  the 
questionnaire  and  field  procedures. 

3.  Postage,  reproduction,  and  printing  costs  for  questionnaires  and 
related  administrative  materials. 

4.  Labor  and  material  costs  for  monitoring  the  data  collection  and 
follow-up  with  nonrespondtnts. 

b.  Labor  and  material  costs  for  editing,  coding,  and  processing 
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(keypunching  or  optical  scanning)  the  information  from  the 
questionnaires  onto  computer  tape. 

6.  Cost  of  spot-checking  to  assure  the  quality  of  the  editing, 
coding,  and  data  processing. 

7.  Cost  of  cleaning  the  final  data  tapes,  that  is,  checking  the 
tapes  for  inconsistent  or  impossible  answers. 

8.  Programming  costs  for  preparing  tabulations  and  special  analyses 
of  the  data. 

9.  Computer  time  for  the  various  tabulations  and  analyses. 

10.  Labor  time  and  material  cost  for  documentation  and 

report  preparation. 

In  addition  to  the  administrative  and  technical  support  provided  by 
the  Office  of  the  Deputy  Assistant  Secretary  of  Defense  (Reserve 
Affairs),  the  Army  Reserve  components,  and  the  Defense  Manpower  Data 
Center,  the  approximate  level  of  support  provided  by  Rand  analytic  and 
survey  staff  was  2.5  person  years.  The  1979  Reserve  Force  Studies 
Surveys  took  more  than  a  year  from  initial  survey  planning  to  having 
results  ready  for  analysis. 
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III.  ORGANIZATION  AND  HANAGEMLNT  OF  FIELDWORK 


BACKGROUND 

Past  survey  data  collection  efforts  with  military  personnel  have 
been  implemented  with  little  or  no  direct  monitoring  of  the  actual  field 
administration  of  the  questionnaires.  Although  frequently  absent, 
monitoring  of  the  survey  administration  is  a  critical  part  of  any  data 
collection  activity.  The  early  identification  and  resolution  of  field 
problems,  particularly  nonresponse  problems,  allows  immediate  follow-up 
using  a  variety  of  techniques  to  increase  survey  response  rates.  Since 
response  rates  for  previous  surveys  with  the  Reserve  Force  population 
have  been  very  low,  field  monitoring  of  the  survey  administration  was 
crucial  in  our  attempts  to  increase  survey  response  rates. 

To  insure  that  the  data  collected  in  the  1979  Reserve  Force 
Studies  Surveys  would  meet  professional  research  standards  and  be  of 
maximum  usefulness  for  policy  analysis,  we  implemented  procedures  for 
on-going  survey  accountability  during  the  seven-month  fieldwork  period. 
Rand  established  procedures  for  monitoring  the  fieldwork  so  that 
unit-level  personnel  would  have  a  sense  of  direct  responsibility  for 
completing  and  returning  the  surveys.  Even  with  the  high  level  military 
support  provided  by  the  Office  of  the  Deputy  Assistant  Secretary  for 
Defense  (Reserve  Affairs)  and  the  Army  Reserve  components,  the  Rand 
survey  staff  felt,  on  the  basis  of  previous  DoD  survey  experience,  that 
some  units  might  give  low  priority  to  the  data  collection  or  fail  to 
respond.  We  also  anticipated  that  some  units  might  not  be  able  to 
administer  the  surveys  during  regular  drills  in  November  and  December 
1979,  as  scheduled.  Since  we  expected  that  most  units  would  receive 
their  survey  materials  the  last  week  in  November  1979,  when  most  drills 
had  been  completed,  we  realized  that  the  survey  administration  might  not 
be  completed  during  the  one  remaining  drill  assembly  in  December  1979. 

We  also  expected  some  reporting  lags  because  of  operational  pressures  at 
the  unit  level,  such  as  emergency  call-ups,  special  maneuvers,  and  unit 
inspections,  which  might  introduce  irregularities  into  unit  activities 
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and  thereby  cause  delays  in  the  survey  administration.  Because 
reservists  typically  drill  with  their  units  only  one  weekend  per  month, 
and  the  survey  administration  was  planned  during  the  Christmas  season, 
Rand  felt  that  many  units  would  need  time  in  early  1980  to  complete  the 
survey.  We  also  anticipated  that  units  with  low  monthly  drill  attendance 
would  need  several  drill  assemblies  (i.e.,  2  to  3  months)  to  administer 
the  Enlisted  Personnel  surveys  to  all  unit  members. 

We  found  that  slightly  less  than  half  of  all  sampled  units  (about  46 
percent)  were  able  to  complete  the  survey  administration  by  December 
1979.  The  remaining  54  percent  needed  additional  drill  time  in  1980  to 
administer  all  of  the  surveys.  Consequently,  we  first  extended  the 
survey  period  to  May  and  then  to  June  1980  to  encourage  survey  partici¬ 
pation.  This  schedule  enabled  about  91  percent  of  all  sample  units  to 
participate. 

From  December  1979  through  June  1980,  Rand's  monitoring  activities 
focused  on  the  early  identification  of  nonparticipating  units  and  the 
development  and  implementation  of  appropriate  follow-up  strategies  to 
encourage  unit  participation.  Other  equally  important  objectives  of  our 
monitoring  activities  were  the  following: 


o  To  identify  problems  encountered  in  the  distribution  and  return 
of  questionnaires  and  to  solve  those  problems  in  a  timely 
fashion. 

o  To  obtain  preliminary  indicators  of  survey  response  rates  and 
the  success  of  the  field  administration  procedures  (that  is, 
whether  sample  units  were  fulfilling  the  administrative 
requirements) . 

o  To  account  for  all  returned  materials  and  prepare  completed 
questionnaires  for  data  processing. 

o  To  obtain  data  needed  to  produce  regular  survey  progress 
reports. 

We  discuss  the  specific  field  monitoring  activities  implemented  for 
this  study  and  how  they  were  organized  below. 


QUESTIONNAIRE  DISTRIBUTION 


Shipment  Verification  Procedures 

The  first  phase  of  our  field  monitoring  was  to  verify  that  all 
sample  units  received  the  survey  materials  sent  to  them  by  the  Intran 
Corporation  through  the  United  Parcel  Service  (UPS).*  The  selection  of  a 
shipment  method  was  a  critical  part  of  the  initial  survey  planning, 
since  we  wanted  to  minimize  the  possibility  of  any  lost  or  undelivered 
shipments  and  guarantee  their  delivery  in  time  for  a  unit  administration 
in  November  or  December  1979,  as  originally  scheduled. 

To  verify  receipt  of  survey  materials  by  all  sample  units  Rand 
implemented  the  following  procedures: 

1.  At  the  time  survey  materials  were  delivered  to  sample  units,  a 
UPS  Acknowledgment  of  Delivery  Form  was  signed  by  the  person 
receiving  the  package  and  was  returned  directly  to  the  UPS 
carrier.  UPS  mailed  these  forms  directly  to  DMDC  for  later 
transmittal  to  Rand.  Within  2  to  5  aays ,  Rand  had  documented 
evidence  of  the  date  materials  were  actually  delivered  to  the 
unit  and  the  name  of  the  person  who  had  accepted  those 
materials.** 

2.  Each  unit  was  also  responsible  for  returning  an  Acknowledgment 
Postcard  directly  to  DMDC  to  verify  receipt  of  the  survey 
materials  and  to  indicate  the  date  the  unit  planned  to 
administer  the  surveys.  The  postage-paid,  preaddressed  postcard 
was  enclosed  in  the  box  with  the  survey  materials.  Thus, 
returned  postcards  indicated  that  the  unit  had  received  the 
shipment  and  reviewed  its  contents.  Immediately  upon  receipt  of 
these  postcards,  DMDC  forwarded  them  to  Rand. 

3.  Rand  reviewed  both  the  returned  UPS  Acknowledgment  of  Delivery 
Forms  and  the  Unit  Acknowledgment  Postcards  to  determine 


•Based  on  our  past  survey  experience,  UPS  is  the  safest,  quickest, 
and  most  cost  efficient  way  of  handling  bulk  mailing  for  a  large-scale 
survey.  For  this  study,  approximately  5>00  boxes  of  questionnaires  were 
distributed  nationwide  to  4*41  sample  units.  To  speed  shipments  to 
distant  areas,  UPS  sent  selected  packages  by  air  freight  instead  of 
truck. 

**UPS  shipment  verification  cost  an  additional  $.34  per  box:  this 
special  service  is  not  automatically  provided  but  must  be  requested  by 
tne  sender. 
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whether  all  units  had  verified  receipt  of  their  materials.  In 
most  cases,  we  received  both  verification  forms  confirming 
shipment  receipt;  in  a  limited  numbe  '  .  cases,  only  the  UPS 
form  was  returned.  We  found  this  form  to  be  the  more  accurate 
verification,  because  UPS  delivery  required  a  unit  signature. 

If  we  received  either  a  UPS  form  or  a  unit  acknowledgment  card, 
we  assumed  that  the  unit  had  received  its  survey  materials. 
Ninety-five  percent  of  all  units  returned  at  least  one  shipment 
verification  form.  The  20  units  (about  5  percent  of  the  total 
sample)  that  did  not  return  a  verification  form  were  the  focus 
of  Rand's  initial  follow-up  activities. 

4.  Rand  survey  staff  attempted  to  contact  the  20  nonresponding 
units  by  calling  the  unit  points-of-contact  designated  by  the 
unit  commander.  If  Rand  was  unable  to  reach  the  unit  directly, 
we  asked  our  points-of-contact  at  the  Army  National  Guard  and 
the  Army  Reserve  to  verify  shipment  receipt  through  military 
channels.  The  Reserve  units  were  contacted  by  our  contact  at 
U.S.  Army  Forces  Command  (the  tORSCOM  liaison  officer  at  Fort 
McPherson).  For  the  Army  National  Guard,  our  contact  at  the 
appropriate  State  Adjutant  General's  Office  contacted  the  unit 
to  verify  receipt  of  the  survey  materials.  We  verified  that  18 
of  the  20  nonresponding  units  had  received  their  materials,  but 
had  failed  to  return  the  acknowledgment  postcards.  The  UPS 
forms  for  these  units  were  apparently  lost  in  the  return 
mailing.  The  two  remaining  units  indicated  that  materials  had 
not  been  delivered.  V<e  discovered  that  materials  for  these 
two  units  had  been  returned  to  the  mailing  contractor  because  of 
insufficient  address  information.  The  correct  mailing  address 
was  obtained  from  the  unit  and  sent  to  the  contractor,  who  then 
forwarded  the  survey  boxes.  With  the  resolution  of  these  two 
address  problems,  we  had  verified  that  all  sample  units  had 
received  their  materials. 

Time  Required  For  Surveys  to  Reach  Sampled  Units 

To  determine  whether  sampled  units  actually  received  their  materials 

in  time  for  the  planned  administration  in  November  and  December  1979,  we 
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tabulated  receipt  dates  from  the  UPS  Acknowledgment  of  Delivery  cards. 
Table  1  breaks  down  the  shipping  time  for  the  survey  materials. 


Table  1 

SHIPPING  TIME  FOR  SURVEY  MATERIALS 


Percentage  of 


21?  Army 
Reserve 
Units 


22k  Army 
National 
Guard  Units 


Receiving  shipment  in 

4  days  or  less 

0.0 

0.0 

5  to  10  days 

79.7 

19.7 

11  to  15  days 

9.2 

58.4 

16  to  2k  days 

3.7 

7.0 

26  or  more  days 

1.8 

1.6 

Unknown  number  of  days 

5.5 

13.4 

NOTE:  Percentages  do  not  add  up  to  100 
because  of  rounding. 


The  figures  in  Table  1  are  based  on  the  assumption  that  shipments 
of  survey  materials  went  out  to  Reserve  units  on  November  19  and  to 
National  Guard  units  on  November  21,  1979.  UPS  delivery  dates  were 
missing  for  42  units — about  10  percent  of  those  sampled — units  that  had 
confirmed  receipt  (by  telephone  or  postcard)  but  had  failed  to  provide 
the  date  of  delivery. 

Vie  had  expected  UPS  delivery  to  take  2  to  7  days;  actually  only 
about  49  percent  of  the  shipments  were  delivered  within  that  period.* 
Most  Reserve  units  received  their  survey  materials  within  10  days;  most 
National  Guard  units  received  theirs  within  2  to  3  weeks.  None  of  the 
shipments  arrived  in  time  to  be  administered  in  November,  and  only  a 
few  arrived  in  time  for  December  administration. 


•One  possible  explanation  for  the  slow  delivery  is  that  boxes  for 
units  that  had  post  office  box  numbers  rather  than  street  addresses 
could  not  be  delivered  directly  to  the  units,  but  had  to  be  picked  up 
by  the  units  at  a  central  post  office,  often  up  to  a  week  later.  Second, 
our  past  experience  with  UPS  suggests  that  large  bulk  mailing  involving 
20-  to  50-pound  boxes  takes  several  days  longer  than  the  normal  2-  to 
7-day  UPS  delivery  time  for  smaller  packages.  Nevertheless,  UPS 
offered  several  advantages:  very  low  loss  rate  for  survey  packages; 
verification  of  delivery  within  3  days;  and  immediate  return  of 
undeliverable  boxes  to  Intran  for  address  correction. 
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QUESTIONNAIRE  RETURNS 


Initial  Return  of  Survey  Materials 

In  early  December  1979,  most  sample  units  returned  acknowledgment 
postcards  to  DMDC  to  verify  receipt  of  materials  and  to  indicate  the 
planned  survey  administration  date(s).  At  this  time,  most  units 
indicated  that  they  planned  to  administer  the  survey  during  January 
drills;  a  few  indicated  administration  during  a  February  drill.  Under 
this  assumption,  we  expected  that  most  completed  surveys  would  be 
returned  to  DMDC  within  1  to  4  weeks  from  the  completion  of  the  unit 
administration,  that  is,  by  the  end  of  February  1980.  Table  2  below 
shows  the  number  of  weeks  required  for  completed  surveys  to  be  returned 
by  sampled  units. 

We  had  expected  that  most  completed  surveys  would  be  returned  within 
12  weeks,  from  December  1979  to  February  1980;  actually,  roughly  70  per¬ 
cent  of  all  Army  Reserve  units  and  only  64  percent  of  all  Army  National 


Table  2 

RETURN  TIME  FOR  COMPLETED  SURVEYS 


Percentage  of 

217  Army  224  Army 
Reserve  National 
Units  Guard  Units 


Returning  survey  materials  in 

1  to  4  weeks 

22.6 

19.2 

5  to  8  weeks 

28. 1 

23.7 

9  to  12  weeks 

20.3 

21.4 

13  to  16  weeks 

8-3 

4.9 

17  to  20  weeks 

2.8 

7.1 

21  to  24  weeks 

4. 1 

6.7 

25  to  28  weeks 

5.5 

7.1 

29  to  32  weeks 

0.5 

0.0 

Unknown  (a) 

0.5 

2.7 

Not  returning  survey  materials 

8.3 

7.1 

NOTE;  Percentages  do  not  add  up  to  100  because  of 
round ing. 

(a)  Includes  shipments  of  completed  nu  cys  that 
were  received  but  erroneously  not  logged  in  at  DMDC; 
thus,  correct  dates  were  not  recorded  for  these  units. 


Guard  units  returned  questionnaires  within  that  period,  by  the  end  of 
February  1980,  we  had  still  received  no  survey  materials  from  about  195 
units,  an  overall  unit  nonresponse  rate  of  33  percent.  In  early  March, 
we  met  with  our  primary  contacts  from  the  Office  of  the  Deputy  Assistant 
Secretary  of  Defense  (Reserve  Affairs),  the  National  Guard  Eureau,  and 
Office  of  the  Chief,  Army  Reserve,  to  discuss  appropriate  follow-up 
strategies  for  encouraging  unit  participation  in  the  survey.  Agreeing 
that  high-ranking  military  personnel  could  more  effectively  than  Rand 
personnel  urge  nonparticipating  units  to  complete  their  questionnaires, 
we  decided  that  the  Rand  survey  staff  would  coordinate  follow-up  letters 
and  telephone  inquiries  from  our  command  contacts  at  FORSCOM  (the  Reserve 
liaison)  and  the  state  adjutants  general  (the  Guard  liaison)  to  nonpar¬ 
ticipating  units.  We  also  decided,  based  on  the  March  questionnaire 
returns,  that  two  additional  drill  sessions  (in  April  and  May)  would 
probably  be  needed  to  enable  the  remaining  units  to  complete  the  survey 
administration. 

In  late  March,  hand  survey  staff  called  our  FORSCOM  contact  and  the 
appropriate  state  adjutants  general  to  inform  them  of  the  current  status 
of  the  survey  and  to  solicit  their  cooperation  in  contacting  nonparti¬ 
cipating  units.  A  letter  confirming  those  telephone  conversations  was 
sent  to  each  of  these  service  contacts  (see  Appendix  E) .  In  early  April 
1980,  FORSCOM  sent  a  letter  of  inquiry  to  each  nonparticipating  Army 
Reserve  unit  (see  Appendix  b) .  The  offices  of  the  involved  state 
adjutants  general  either  sent  letters  or  called  nonparticipating  Guard 
units  directly.  Both  the  Reserve  and  Guard  contacts  informed  all  195 
nonparticipating  units  that  the  survey  completion  date  had  been  extended 
to  May  31.  1980,  stressed  the  importance  of  the  study,  and  encouraged  the 
units  to  participate. 

As  soon  as  the  military  follow-up  inquiries  were  completed,  the 
service  contacts  sent  Rand  an  updated  report  of  the  survey  status  of  each 
nonparticipating  unit.  In  addition  to  the  military  follow-up,  Rand 
survey  staff  called  at  random  about  one  third  of  the  nonparticipating 
units  (1)  to  verify  that  the  unit  had  been  notified  by  either  FORSCOM  or 
an  adjutant  general's  office  of  the  extension  of  the  survey  period  and 
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had  been  encouraged  to  participate  in  the  survey  and  (2)  to  obtain  a 
preliminary  indication  of  the  unit's  probable  response  to  the  official 
follow-up  attempts.  The  Rand  follow  up  found  that  some  units  had  already 
completed  and  returned  their  questionnaires,  others  had  completed  some 
but  not  all  of  the  questionnaires,  and  six  Reserve  units  had  never 
received  the  survey  materials.  Since  UPS  delivery  forms  showed  that 
these  materials  had  been  delivered  and  accepted  by  someone  at  the  unit, 
we  suspected  that  the  materials  had  been  misplaced  or  discarded.  In  any 
event,  we  sent  a  second  shipment  of  surveys  to  each  of  these  six  units 
and  obtained  the  cooperation  of  four  of  them  in  completing  the  survey. 

Results  of  Following  Up  Nonparticipating  Units 

Table  2  shows  the  success  of  our  initial  follow-up  attempts  with 
nonparticipating  units.  Some  76  percent  of  all  sampled  Army  National 
Guard  units  had  returned  completed  surveys  after  20  weeks,  and  an 
additional  7  percent  after  24  weeks;  however,  this  means  that  after  24 
weeks,  17  percent  of  the  Guard  units  still  had  not  returned  any  surveys 
materials.  In  the  Army  Reserve,  82  percent  of  all  units  had  returned 
completed  surveys  after  20  weeks,  and  an  additional  4  percent  after  24 
weeks.  Thus,  after  24  weeks,  15  percent  of  the  Army  Reserve  units  still 
had  not  returned  any  survey  materials. 

Although  our  initial  follow-up  attempts  succeeded  in  increasing  the 
overall  unit  participation  rate  from  67  percent  to  about  84  percent,  we 
were  still  concerned  that  16  percent  of  the  sample  units  had  not  returned 
any  surveys  after  24  weeks.  In  late  May,  we  decided  to  implement  one 
final  follow-up  strategy  to  obtain  at  least  one  form,  the  Unit  Survey, 
from  nonparticipating  units.  Rand  sent  a  final  letter  to  those 
nonresponding  units  and  solicited  their  cooperation  in  completing  and 
returning  the  Unit  Form.  (See  Appendix  E  for  a  copy  of  this  final 
follow-up  letter.)  Units  which  had  indicated  that  survey  materials  were 
returned  prior  to  the  end  of  May  were  asked  to  fill  out  a  second  Unit 
Form,  since  the  earlier  materials  may  have  been  lost.  The  survey  end 
date  was  extended  once  again  and  all  survey  materials  were  expected  at 
DM  DC  no  later  than  June  30,  1980.  Thanks  to  the  final  follow-up  efforts 
in  May,  Rand  obtained  completed  questionnaires  from  half  of  the  units 
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that  had  failed  to  return  them  during  the  first  24  weeks,  reducing  the 
nonresponse  rate  from  16  percent  to  8  percent.  Table  2  shows  that  by  the 
end  of  the  survey  period,  DMDC  had  received  some  or  all  survey  materials 
from  93  percent  of  all  Army  National  Guard  units  and  92  percent  of  all 
Army  Reserve  units.  Response  rates  to  each  of  the  four  questionnaires 
fielded  for  this  study  are  discussed  in  Section  IV. 

Incomplete  Shipments 

Some  40  units  returned  shipments  without  Unit  and/or  Commander 
forms;  in  all,  50  forms  were  missing.  Whereas  the  nonresponse  of  an 
individual  in  the  Enlisted  Personnel  Survey  lowered  the  overall  response 
rate  only  slightly,  nonresponse  for  the  Unit  and/or  Commander  Survey 
would  have  greatly  limited  our  analytic  capacity,  since  we  sought  to 
merge  data  about  the  characteristics  of  the  unit  and  the  commanding 
officer  with  each  of  the  Enlisted  Personnel  Surveys.  Therefore,  in  early 
April  1980,  Rand  sent  follow-up  letters  to  the  40  units  which  had 
returned  incomplete  shipments.  This  letter  thanked  the  unit  for  its 
participation,  indicated  that  the  Unit  and/or  Commander  forms  were 
missing  from  the  returned  shipment,  stressed  the  importance  of  the 
missing  forms,  and  requested  that  units  complete  the  remaining  form(s)  by 
May  31,  1980.  Another  copy  of  the  missing  form(s)  was  enclosed  with  the 
letter  (see  Appendix  E) . 

In  response  to  Rand's  request,  31  of  the  40  units  contacted  returned 
a  total  of  44  Unit  and/or  Commander  forms.  The  final  response  rate  to 
the  Unit  and  Commander  Surveys  was  encouragingly  high — about  80  percent 
in  each  component.  We  attribute  this  improvement  to  effective  follow-up, 

EDITING  PROCEDURES 

All  of  the  completed  questionnaires  and  accompanying  survey 
materials  were  returned  by  service  administrators  at  the  441  sample  units 
directly  to  the  Defense  Manpower  Data  Center  (DMDC)  over  the  seven  months 
from  December  1979  through  June  1980.  The  large  volume  of  survey 
materials  expected  at  DMDC  required  exact  document  control  and  sample 
accountability  procedures.  The  following  document  receipt  and  editing 
procedures  were  developed  by  Rand  and  implemented  by  DM DC  personnel: 
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o  Maintaining  a  Shipment  Receipt  Log  containing  the  date 
materials  were  received  at  DMDC. 

o  Completing  for  each  returned  box  of  surveys  a  Document  Control 
Form  (DCF)  containing  the  following  information: 

-  Record  Control  Number  (RCN),  an  administrative  ID 
assigned  by  DMDC  to  each  sampled  unit 

-  Unit  location  (City,  State) 

-  Component  ID  (Army  Reserve  or  Army  National  Guard) 

-  Unit  Identification  Code  (UIC) 

-  Date  shipment  was  received  at  DMDC 

-  Number  of  assigned  L1-E9  unit  personnel,  tabulated  from 
returned  Unit  Survey 

-  Number  and  type  of  completed  questionnaires  returned 
to  DMDC 

-  Number  of  questionnaires  marked  "Refused" 

-  Initials  of  the  DMDC  clerk  who  completed  the  Document 
Control  Form 

-  Indicator  of  whether  the  unit  roster,  showing  which  unit 
members  had  participated  in  the  survey,  had  been  returned. 

o  Editing  and  preparing  the  Enlisted  Personnel  Surveys  for  ship¬ 
ment  to  the  optical  scanning  contractor  for  initial  processing, 
including: 

-  Entering  the  RCN  on  each  completed  questionnaire  and 
comment  sheet(s)  returned  by  enlisted  personnel 

-  Mailing  (by  certified  delivery)  edited  enlisted  personnel 
surveys,  in  batches,  separated  by  form  type,  to  the  optical 
scanning  contractor  for  initial  processing 

-  Maintaining  shipment  records  of  the  number  of  question¬ 
naires,  by  form  type,  shipped  to  the  optical  scanning 
contractor  and  the  actual  shipment  dates. 

o  Providing  Rand  with  copies  of  (1)  completed  Document  Control 
Forms,  comment  sheets  returned  by  enlisted  personnel,  and 
completed  Unit  and  Commander  Surveys  (on  a  daily  basis); 

(2)  the  DMDC  Shipment  Receipt  Log  of  returned  questionnaires 


(on  a  daily  basis);  and  (3)  the  DMDC  log  of  completed  surveys 
sent  to  the  optical  scanning  contractor  (immediately  upon 
shipment  of  materials). 


The  data  produced  as  part  of  the  DMDC  document  control  procedures  were 
used  by  Rand  to  prepare  regular  Survey  Progress  Reports,  discussed  below 
in  the  subsection  Survey  Reporting  Requirements. 

Editing  of  Unit  and  Commander  Surveys 

After  the  completed  Unit  and  Commander  Surveys  were  inventoried, 
logged  in,  and  given  a  RCN  at  DMDC,  they  were  sent  each  day  to  Rand  for 
editing.  The  objectives  of  the  questionnaire  edit  were  as  follows; 

o  To  prepare  questionnares  for  data  entry  by  checking  for 

legibility,  assigning  missing  value  and  other  audit  codes, 
zero-filling  numeric  fields,  and  rounding  time  and  income 
entries  as  needed. 

o  To  assign  numeric  codes  to  respondents'  handwritten  entries 
in  answer  to  questions  ragarding,  for  example,  annual  training 
bases  and  recent  training  site  with  active  army  units. 

o  To  review  marginal  comments  provided  by  respondents  and,  where 
appropriate,  incorporate  them  into  the  data  by  generating  new 
response  codes;  otherwise,  to  document  them  in  a  format  for 
later  review  by  the  analysts. 

o  To  identify  any  questions  for  which  the  respondent  marked  more 
than  one  answer  but  for  which  only  one  answer  was  required.  A1 
such  problems  had  to  be  resolved  by  survey  and  analytic  staff 
prior  to  data  entry. 

Each  Rand  editor  was  responsible  for  proofreading  the  one  Commander 
Survey  and  one  Unit  Survey  from  each  sample  unit,  using  a  set  of 
question-by-question  editing  specifications  developed  by  the  Rand-DoD 
Survey  Group.  Ihe  primary  purposes  of  the  edit  were  to  verify  that  all 
information  had  been  properly  recorded  in  the  two  questionnaires  and  to 
ensure  that  the  documents  could  be  directly  keypunched  at  DMDC. 
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Editing  of  Enlisted  Personnel  Surveys 

After  incoming  Enlisted  Personnel  Surveys  were  inventoried,  logged 
in,  and  given  an  RCN,  they  were  reviewed  for  scannability .  The  two 
Enlisted  Personnel  Questionnaires,  Forms  1  and  2,  were  designed  as 
self-administered  forms  suitable  for  subsequent  processing  through 
optical  scanning.  DMDC  personnel  reviewed  all  completed  surveys  to  make 
sure  that  the  questionnaires  did  not  contain  stray  markings,  that  pencil 
markings  were  dark  enough  to  be  scanned ,  and  that  questionnaires  which 
had  been  marked  with  ink  or  which  contained  torn  or  missing  pages  were 
identified.  Inked  questionnaires  and  other  potentially  unscannable 
questionnaires  were  subsequently  remarked  with  scannable  pencils. 

Upon  completion  of  editing,  DMDC  mailed  Forms  1  and  2  to  Intran,  the 
optical  scanning  contractor,  for  the  preparation  of  machine-readable 
data.  To  simplify  processing  for  Intran,  DMDC  sorted  the  questionnaires 
by  form  number  prior  to  shipping. 

SURVEY  REPORTING  REQUIREMENTS 

As  part  of  our  overall  monitoring  strategy,  we  produced  regular 
Survey  Progress  Reports  for  the  Office  of  the  Deputy  Assistant  Secretary 
of  Defense  (Reserve  Affairs)  and  our  numerous  contacts  at  the  Army 
Reserve  components.  These  survey  reports  helped  us  to  keep  in  touch  with 
our  service  contacts  throughout  the  fieldwork  period.  They  also  provided 
preliminary  indicators  of  field  response  rates  and  identified  potential 
fieldwork  problems  so  that  appropriate  follow-up  strategies  could  be 
implemented.  Using  data  produced  by  DMDC  as  a  result  of  their  document 
control  activities,  Rand  created  a  computerized  survey  file  for  each 
sample  unit  which  contained  the  following  information: 

1.  Sample-Related  Data,  including  Record  Control  Number,  unit 
address,  designation  and  prefield  estimates  of  the  number  of 
E1-E9  personnel  assigned  to  unit. 

2.  Survey  Data  from  DMDC  Field  Monitoring  Forms,  including 

-  Date  materials  were  delivered  to  unit  (UPS  dates) 

-  Date  materials  were  returned  to  DMDC 
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-  Number  of  E1-E4  and  E5-E9  personnel  assigned  to  unit 
at  the  time  of  the  survey  administration 

-  Number  of  Completed  E1-E4  and  E5-E9  surveys  returned  to 
DMDC 

-  Estimated  unit  response  rate  for  enlisted  personnel  surveys 
(that  is,  the  ratio  of  the  number  of  completed  E1-E9  surveys 
returned  to  the  number  of  assigned  E1-E9  in  unit  at  the  time 
of  survey  administration) 

-  Indicators  of  whether  or  not  the  unit  returned  the  Unit  and 
Commander  forms 

-  Indicators  of  whether  or  not  the  unit  returned  the  unit 
roster  showing  which  members  had  participated  in  the 
survey  and  reasons  for  individual  nonparticipation  (such  as 
absent  from  drills,  at  school,  and  refused  to  participate). 

Rand  produced  a  Fieldwork  Report  (every  2-3  weeks)  which  identified 
the  units  that  had  and  had  not  returned  materials.  For  units  which  had 
returned  surveys,  the  number  and  type  of  forms  returned  and  indicators  of 
missing  forms  were  also  included  in  the  report.  Units  which  had  not 
returned  any  surveys  were  isolated  for  later  follow-up  attempts.  Summary 
statistics  from  the  unit  profiles  of  survey  participation  were  abstracted 
and  circulated  to  our  DoD  client  and  the  Army  Reserve  components.  In 
addition  to  the  formal  survey  reports,  we  provided  informal  reports  to 
our  contacts  at  the  Army  Reserve  and  Army  National  Guard. 
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IV.  ASSESSING  THE  FIELD  EXPERIENCE 


This  section  assesses  the  success  of  the  survey  administration 
procedures  developed  for  the  1979  Reserve  Force  Studies  Surveys.  The 
evaluation  of  data  collection  procedures  is  approached  from  several 
directions.  First,  we  examine  field  results  through  measures  of  survey 
response  rates.  Second,  the  response  rate  obtained  by  the  present 
reserve  survey  are  compared  with  results  for  similiar  military  and 
civilian  studies.  Third,  we  examine  the  adherence  by  individual  units  to 
the  survey  schedule  and  administrative  requirements. 


SURVEY  RESPONSE  RATES 

The  most  commonly  used  measure  of  success  or  failure  in  the  field  is 
the  survey  response  rate.  The  response  rates  for  this  study  were 
computed  as  the  number  of  questionnaires  completed  by  eligible  sample 
members  divided  by  the  total  number  of  eligible  sample  members.  These 
calculations  followed  the  formula  suggested  in  the  American  Statistical 
Association  report  on  methods  for  assessing  survey  practices  (Bailar  and 
Lanphier,  1978).  Using  this  approach,  we  calculated  the  response  rate 
for  the  surveys  of  enlisted  personnel  as  the  ratio  of  the  number  of 
completed  Enlisted  Personnel  Surveys  to  the  number  of  enlisted 
personnel  assigned  to  the  sampled  unit  at  the  time  of  the  survey 
administration.*  For  the  Unit  and  Commander  Surveys,  response 
rates  were  calculated  as  the  number  of  sampled  units  for  which 
completed  questionnaires  were  received  divided  by  the  total  number 
of  sampled  units. 

Overall  survey  response  rates  to  each  of  the  four  questionnaires 
developed  for  this  study,  using  the  procedures  discussed  above,  are 


*The  data  pertaining  to  the  number  of  enlisted  personnel  who  were 
members  of  the  sampled  units  at  the  time  of  the  survey  were  tabulated 
from  the  completed  Unit  Surveys.  Using  unit  records  of  actual 
personnel  strength,' "Wie"  WIT  Commander  recorded  the  required  data  in  the 
Unit  Survey.  For  units  that  did  not  return  completed  Unit  Surveys 
(approximately  18  percent  of  the  ■sample),  the  personnel  data  was 
austracted  From  the  reserve  personnel  files  available  at  the  time  of 
sample  selection,  i.e.,  as  of  July  or  August  1979,  based  on  the  Reserve 
Components  Personnel  Data  System. 
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presented  in  Table  3.  These  response  rates  clearly  show  the  relationship 
between  what  we  wanted  sampled  units  to  do  and  what  actually  happened  in 
the  field.  The  unit  commanders  were  diligent  in  completing  the  two 
questionnaires  which  were  designed  for  their  own  use,  namely  the  Unit 
and  Commander  Surveys.  In  both  the  Army  Reserve  and  the  Army  National 
Guard,  the  completion  rates  for  these  two  surveys  were  over  80  percent, 
well  over  our  projected  response  rate  of  70  percent  for  all  surveys.  The 
number  of  returned  questionnaires  from  enlisted  unit  members,  however, 
was  lower  than  the  projected  70  percent:  65  percent  for  senior  enlisted 
personnel  (E5-E9)  and  under  49  percent  for  junior  enlisted  personnel 
(E1-E4).*  The  completion  rate  of  53  to  67  percent  for  Army  National  Guard 
enlisted  personnel  was  slightly  higher  than  the  44  to  63  percent  obtained 
from  Army  Reserve  enlisted  personnel. 


Table  3 

RESPONSE  RATES  FOR  SURVEY  FORMS 


Survey  Form 

Army 

National  Guard 

Army 

Reserve 

Total 

Sample 

Size 

Average 

Response 

Rate 

Sample 

Size 

Response 

Rate 

Sample 

Size 

Response 

Rate 

Unit  (a) 

224 

80.8% 

217 

83.4% 

441 

82.1% 

Commander  (a) 

224 

79.9% 

217 

82.5% 

441 

81.2% 

E5  to  E9  (b) 

8,722 

66.9% 

9,099 

63.2% 

17,821 

65.1% 

El  to  E4  (b) 

11,341 

52.8% 

10,445 

44.2% 

21,786 

48.5% 

(a)  Sample  size  is  the  number  of  units  included  in  the  survey. 

(b)  For  units  that  returned  completed  Unit  Surveys,  the  sample 
size  is  the  number  of  enlisted  personnel  in  ?SCTi  psy  grade  group 
reported  in  the  survey.  For  units  that  did  not  return  completed 
Unit  Surveys,  sample  size  is  based  on  the  number  of  enlisted 
personnel  in  each  pay  grade  group  at  the  time  of  sample  selection, 
i.e.,  as  of  July  or  August  1979,  based  on  the  Reserve  Components 
Personnel  Data  System. 


•Unfortunately,  the  response  rate  to  the  Enlisted  Personnel  Surveys 
was  depressed  by  about  4  percent  as  a  result  of  the  loss  in  the  U.S.  mail 
of  five  boxes  containing  approximately  1500  completed  questionnaires  that 
were  being  sent  from  DMuC  in  Alexandria,  Virginia,  to  the  optical  scan 
contractor,  Intran,  in  Minneapolis,  Minnesota. 
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Variation  by  Unit 

An  alternative  way  to  look  at  response  rates  is  to  examine  the 
unit-specific  response  rates  for  the  Enlisted  Personnel  Surveys.  Our 
analysis  of  fieldwork  data  shows  that  response  rates  among  enlisted 
personnel  varied  greatly  among  sampled  units,  from  90  to  100  percent 
individual  participation  to  less  than  10  percent.  A  high  unit  response 
rate  indicates  that  the  unit  commander  administered  the  personnel  surveys 
to  a  large  percentage  of  the  enlisted  unit  members,  with  few  refusals  and 
absences.  Lower-than-expected  unit  response  rates  may  have  resulted  from 
individual  refusals  to  participate  or  low  attendance  at  monthly  Reserve 
unit  drills.  The  existence  of  "split"  Reserve  units  may  also  have  had  a 
negative  effect  on  response  rates.  We  found  that  roughly  20  percent  of 
the  units  selected  for  sampling  were  organized  into  two  or  more  sections, 
which  drilled  at  different  times  and  in  different  geographical  locations. 
Enlisted  response  rates  in  these  split  units  may  have  been  lower  because 
the  unit  failed  to  administer  the  surveys  to  enlisted  members  at  all  unit 
drill  locations.  The  relatively  low  response  rates  for  junior  enlisted 
members  depressed  the  overall  response  rates  for  unit  enlisted  personnel. 

Most  units  obtained  somewhat  better  than  average  response  rates. 

The  mean  enlisted  response  rate  for  Army  National  Guard  units  was  about 
60  percent  and  for  Army  Reserve  units,  54  percent. 

The  close  distribution  of  unit  response  rates  for  the  Army  National 
Guard  and  the  Army  Reserve  indicates  that  there  were  no  significant 
service  differences  in  the  survey  administration.  For  92  National  Guard 
units,  or  41  percent  of  those  sampled,  we  achieved  our  projected  response 
rate  of  70  percent  or  higher.  More  impressive  is  the  fact  that  more  than 
half  the  Guard  units  in  this  group  (that  is,  50  units)  returned  completed 
surveys  for  well  over  80  percent  of  all  El  to  E9  personnel.  For  78 
additional  Guard  units,  or  35  percent  of  those  selected,  between  50  and 
69  percent  of  all  enlisted  personnel  returned  completed  surveys.  Thus, 
completion  rates  of  50  percent  or  higher  were  obtained  from  170,  or  76 
percent  of  all  sampled  National  Guard  units. 

In  77  Reserve  units,  or  36  percent  of  those  selected,  we  achieved 
our  projected  response  rate  of  70  percent  or  higher.  Moreover,  31 
Reserve  units  in  this  group  (14  percent  of  the  total  sample)  returned 
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completed  surveys  for  well  over  80  percent  of  all  El  to  E9  personnel. 

For  87  additional  Reserve  units,  or  40  percent  of  those  sampled,  we 
received  completed  surveys  from  between  50  and  69  percent  of  all  unit 
enlisted  personnel.  Thus,  completion  rates  of  50  percent  or  better  were 
obtained  from  164,  or  76  percent  of  all  sampled  Reserve  units. 


Comparisons  with  Other  Military  Studies 

The  response  rates  for  the  1979  Reserve  Force  Studies  Surveys 
compared  favorably  with  rates  for  past  surveys  of  Reserve  Force 
personnel  and  with  similar  surveys  of  the  active  force  population.  It 
was  not  uncommon  for  previous  Reserve  surveys  to  achieve  response  rates 
§  of  20  to  40  percent.  One  reason  for  the  higher  response  rates  achieved 

1  by  the  1979  Reserve  Surveys  may  be  the  unit-based  sampling  and 

administrative  procedures  used  in  this  survey.  Unit  sampling  not  only 
reduced  the  mailing  and  administrative  costs  of  conducting  the  survey, 
but  it  also  simplified  respondent  follow-up  procedures  to  ensure  higher 
^  survey  response  levels.  We  had  projected  that  such  a  technique  would 

result  in  response  rates  between  50  and  70  percent. 

The  present  and  previous  Reserve  studies  were  so  different  as  to 
f 

I  make  detailed  comparisons  of  response  rates  across  surveys  impossible.  A 

\  more  appropriate  comparison  is  the  rates  obtained  from  two  recent  studies 

of  the  active  force  population  using  generally  similar  survey  procedures: 
v  the  1978  DoD  Survey  of  Officers  and  Enlisted  Personnel*  and  the  1981 

Variable  Housing  Allowance  Survey.**  In  fact,  the  1978  DoD  Survey 
►  served  as  a  model  for  both  the  present  Reserve  study  and  the  1981 

Variable  Housing  Allowance  Survey. 

The  1978  DoD  Survey  of  Officers  and  Enlisted  Personnel  was 
•j  designed  by  Rand  to  collect  data  needed  to  analyze  several  research  and 

policy  areas  ranging  from  reenlistment  behaviors  to  utilization  of  women 

t 

i  in  the  armed  forces .  A  mail  surve.  was  administered  from  January  to  June 

<  1979  to  a  worldwide  sample  of  about  93,000  men  and  women  on  active  duty 

’I 

in  all  four  services.  The  1981  Variable  Housing  Allowance  Survey  was 
designed  to  obtain  detailed  information  regarding  the  actual  cost  of 


•See  2.  D.  Doering,  D.  W.  Orissmer,  J.  A.  Hawes,  and  W.  P.  hutzler, 
1978  DoD  Survey  of  Officers  and  Enlisted  Personnel:  Survey  Design  and 

aabi  in  l  strati  vr mrc-eaures,  me  Kanti  rort'oration-,1  y- nres-wm,','  apm — 

••The  results  will  be  reported  by  DoD. 


.’***  4*V-v.'A-V 
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civilian  housing  occupied  by  active  force  personnel  who  receive  a  basic 
Allowance  for  Quarters  ( BAQ ) .  This  data  will  be  used  to  determine  how 
much  of  an  additional  monthly  allowance  should  be  given  to  military 
personnel  to  cover  geographical  variation  in  housing  costs.  Rand 
provided  technical  assistance  to  DoD  in  the  overall  survey  design  and 
development  of  administrative  procedures  and  forms.  The  instrument  was  a 
self-administered  mail  survey  which  was  sent  to  approximately  350,000 
persons  on  active  duty  in  all  four  services.  To  improve  response  rates 
and  sample  accountability,  all  three  studies  used  self-administered  mail 
surveys,  the  same  basic  survey  administrate ve  model,  and  Service  channels 
to  collect  the  data.  Under  this  procedure,  questionnaires  and  survey 
accountability  materials  were  mailed  directly  to  a  service  administrator 
who,  in  turn,  was  responsible  for  the  distribution  and  return  of 
completed  materials.  Unlike  field  operations  for  previous  Reserve  and 
other  DoD  Surveys,  these  three  studies  imposed  stringent  requirements  for 
sample  accountability  on  all  service  administrators  responsible  for  the 
actual  data  collection.  Rand  monitored  and  followed  up  the  data  collec¬ 
tion  (or  provided  guidelines  to  DoD)  to  increase  response  rates. 

The  1978  DoD  Survey  and  the  1981  Variable  Housing  Allowance 
Survey  obtained  field  results  and  response  rate  patterns  quite 
similar  to  those  obtained  in  the  present  Reserve  survey.  The  response 
rate  comparisons  between  the  present  Reserve  study  and  the  two  recent 
active  force  surveys  are  given  in  Table  4. 

Although  the  response  rates  shown  in  Table  4  ranged  from  56  to  82 
percent,  the  differences  appear  between  enlisted  personnel  and  officers, 
rather  than  between  surveys.  The  data  demonstrate  that  response  rates 
increase  with  military  rank.  Thus,  we  obtained  higher  response  rates 
from  officers  (72  to  82  percent)  than  from  enlisted  personnel  (56  to  62 
percent).  Evidence  from  this  study  and  the  1978  DoD  Survey  also 
indicates  that  senior  enlisted  personnel  (grades  E5  to  E9)  respond  at 
higher  levels  than  junior  enlisted  personnel  but  not  as  high  as  officers 
(see  Table  3.  above). 

The  distribution  of  overall  response  rates  among  enlisted  personnel, 
as  shown  in  Table  4,  varied  only  slightly.  The  increase  of  a  few 
percentage  points  in  the  response  rate  for  the  1981  Variable  Housing 


t 
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Table  4 

COMPARISON  OF  RESPONSE  RATES  IN  THREE  MILITARY  SURVEYS 


Sample 

Size 

Response  Rate 

Name  of  Survey 

Enl isted 

Officer 

Enlisted 

Officer 

1979  Reserve  Force  Studies 
Surveys 

39,607 

441 

56  % 

82% 

1978  DoD  Survey  of  Officers 
and  Enlisted  Personnel  (a) 

62,840 

27,215 

58% 

72% 

1981  Variable  Housing  Allow¬ 
ance  Survey  (b) 

241,235 

136,097 

62  % 

74% 

NOTE;  The  administration 

time  for  all 

three 

surveys  was 

one 

hour . 

(a)  Sample  size  and  response  rates  were  reported  in  N-1458-MRAL. 

(b)  Sample  size  and  response  rates  were  provided  to  the  author 
by  the  Defense  Manpower  Data  Center. 


Allowance  Survey  may  have  been  occasioned  by  the  potential  benefits  to 
the  respondents  that  it  offered.  Reason  suggests — and  methodological 
studies  of  mail  survey  response  rates  substantiate  the  view — that  mail 
questionnaires  are  more  likely  to  be  returned  if  the  respondent  believes 
that  doing  so  will  be  to  his  advantage  (Heberlein  and  Baumgartner,  1978). 
The  fact  that  the  actual  response  rate  for  the  housing  Allowance  survey 
was  only  slightly  higher  than  that  for  the  other  military  surveys 
suggests  that  operational  constraints  on  the  data  collection  may  have 
precluded  greater  participation;  that  is,  some  sample  members  who  might 
have  participated  in  the  study  may  never  have  had  the  opportunity  because 
of  the  failure  of  service  administrators  to  distribute  and  collect  some 
questionnaires. 

The  similarity  of  response  rates  in  the  three  military  surveys,  as 
shown  in  Table  4,  should  not  lead  to  the  conclusion  that  response  rates 
for  the  present  study  were  as  easy  to  achieve  as  the  rates  for  the  other 
two  studies.  As  noted  above,  past  Reserve  studies  had  much  lower 
response  rates  than  either  active  forces  studies  or  the  1979  Reserve 
Force  Studies  Surveys.  The  higher  rate  for  this  study  is  due  in  part  to 
the  smaller  sample  size  and  unit-based  sampling  strategy,  which  enabled 
Rand  to  monitor  the  survey  administration  more  closely  than  was  possible 
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in  the  larger  studies.  We  were  able  to  follow  up  nonparticipating  units 
because  we  had  the  names  and  telephone  numbers  of  the  441  service  survey 
administrators  and  we  were  authorized  to  contact  them  directly. 

In  the  larger  studies,  where  over  3000  unit-level  personnel  were 
responsible  for  the  data  collection,  it  was  not  possible,  in  most  cases, 
for  Rand  to  make  direct  contact  with  the  individual  survey  administrators. 
Instead,  we  had  to  coordinate  our  follow-up  requirements  through  a 
primary  point-of-contact  designated  by  and  for  each  service.  This 
operational  organization  for  the  data  collection  and  monitoring  meant 
that  the  actual  survey  administrators  were  largely  anonymous  to  Rand.  We 
did,  however,  closely  monitor  the  questionnaire  returns  from  the  service 
administrators,  and  this  check  enabled  us  to  provide  each  service  with 
up-to-date  information  on  the  status  of  the  data  collection.  At  Rand's 
request,  each  service  PPOC  sent  follow-up  notices  through  military 
channels  to  request  cooperation  among  those  personnel  who  were  inclined 
to  refuse  to  participate. 

Even  though  the  sample  for  the  present  survey  was  considerably 
smaller  than  the  sample  for  the  two  active  force  surveys,  the  field 
period  for  this  survey  was  longer  because  extra  time  was  needed  to 
encourage  survey  participation  among  Reserve  Force  personnel.  The  field 
results  for  the  current  study  indicate  that  advanced  notification  of 
sample  members,  with  successive  follow-ups  and  reminders  (about  four 
additional  contacts)  throughout  the  field  period  produced  response  rates 
comparable  to  those  of  the  other  military  studies. 

Comparisons  with  Civilian  Studies 

Although  as  a  rule  military  and  civilian  mail  surveys  differ 
substantively  in  the  issues  they  address,  they  face  similar  problems  in 
obtaining  high  levels  of  cooperation  among  respondents  who  are  asked  to 
complete  and  return  a  self-administered  questionnaire.  In  designing  the 
1979  Reserve  Force  Studies  Surveys,  we  tried  to  adapt  civilian 
techniques  for  encouraging  survey  participation  to  the  military 
environment.  A  comparison  of  military  and  civilian  response  rates  to 
mail  surveys  will  (1)  indicate  whether  or  not  military  and  civilian 
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cooperation  levels  are  similar  and  (2)  identify  similarities  and 
differences  in  techniques  used  to  encourage  survey  participation. 

Four  major  conclusions  can  be  drawn  from  the  research  literature 
on  response  rate  trends  in  recent  civilian  mail  surveys: 

1.  Significant  variations  in  mail  survey  response  rates  (from  less 
than  20  percent  to  100  percent)  among  investigators,  subject 
populations,  questionnaires,  and  administrative  procedures  make 
simple  generalizations  difficult. 

2.  Low  response  rates  are  a  major  drawback  to  the  use  of  mail 
questionnaires.  Without  extensive  follow-up  of  nonrespondents, 
mail  survey  response  rates  would  average  well  below  50  percent. 
The  more  effective  mail  surveys  use  a  greater  number  of  follow¬ 
ups  and  more  varied  follow-up  techniques,  such  as  telephone 
contact  and  special  mailings.  Thus,  mail  surveys  are  less 
attractive  under  circumstances  where  respondent  follow-up  is 
impossible  or  limited. 

3.  While  many  studies  report  generally  low  response  rates  to  mail 
surveys,  a  growing  number  of  researchers  (Dillman,  1978; 
Heberlein  and  baumgartner,  1978;  and  Kanuk  and  berenson,  1975) 
have  shown  that  high  response  rates  (i.e.,  75  percent  and  above) 
to  mailed  questionnaires  can  be  obtained  if  sufficient  resources 
and  design  skills  are  applied  to  the  data  collection. 

4.  The  reseacher  has  available  and  should  use  a  variety  of 
techniques  to  increase  response  levels  to  mail  surveys, 
including: 

a.  Making  contacts — that  is,  follow-ups  and  reminders 

by  means  of  telephone  calls,  personal  visits,  mailgrams, 
reminder  postcards,  and  special  delivery  letters — to 
encourage  survey  participation.  Each  additional  contact  has 
been  shown  to  bring  added  responses. 

b.  Stressing  the  importance  of  the  survey  and  the  respondent's 
participation  in  it.  Official  letters  (advance  and 
follow-up)  from  respected  sponsors  might  serve  to  motivate 
respondents  to  return  surveys.  The  respondent's  knowledge 
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of  and  affiliation  with  the  signer  of  the  letter  may  tend  to 
increase  his  perception  of  the  importance  of  the  study, 
c.  Minimizing  the  respondent  burden  in  participating  by 
providing  stamped,  self-addressed  return  envelopes  and 
designing  questionnaires  to  be  as  fast  and  easy  as  possible 
to  fill  out. 

The  response  rates  for  the  1979  Reserve  Force  Studies  Surveys 
clearly  fell  between  the  extremes  reported  for  civilian  mail  surveys, 
resembling  most  closely  the  mean  final  response  rate  of  about  61  percent 
found  by  Heberlein  and  Baumgartner  (1978). 

Our  experience  in  conducting  the  current  reserve  study  indicates 
that  much  of  the  civilian  work  on  factors  affecting  response  rates  can  be 
directly  applied  to  military  data  collection  efforts.  Overall  response 
rates  were  positively  correlated  with  the  level  of  the  field  effort. 
Military  personnel,  like  their  civilian  counterparts,  responded  favorably 
to  multiple  follow-up  efforts  by  both  Rand  and  military  officials.  The 
basic  strategies  used  to  encourage  civilian  survey  participation  were 
also  extremely  effective  in  increasing  survey  returns  from  military 
personnel. 

The  major  difference  between  fielding  military  and  civilian  surveys 
involves  the  need  to  obtain  general  administrative  support  irom  military 
officials,  as  well  as  respondent  support,  for  survey  data  collection.  In 
civilian  studies,  efforts  to  encourage  survey  participation  are  targeted 
solely  at  respondents.  In  military  studies,  however,  before  the 
individual  respondent  can  be  approached,  his  organizational  superiors 
must  endorse  the  survey,  authorize  service  personnel  to  participate,  and 
allocate  administrative  resources  to  assist  in  the  data  collection. 

Thus,  efforts  to  improve  military  survey  response  rates  must  be  directed 
at  both  service  leaders  and  individual  respondents. 

The  response  rate  problems  ir.  military  data  collection  are 
complicated  further  by  the  fact  that  the  actual  responsibility  for  the 
survey  administration  is  delegated  to  numerous  service  administrators. 

For  example,  the  1979  Reserve  Force  Studies  Surveys  involved  441 
service  admit  i Jtrators  in  the  survey  administration  and  the  1978  DoD 


Survey  over  3000.  The  service  administrators  were  responsible  for  (1) 
distributing  the  self-administered  questionnaires  to  sampled  members 
under  their  jurisdiction,  (2)  scheduling  a  time  and  place  for  either 
group  or  individual  survey  administration  during  military  working  hours, 
and  (3)  returning  completed  surveys.  They  were  also  responsible  for 
keeping  a  detailed  accounting  of  individuals  who  were  not  available  to 
participate;  without  this  requirement  on  the  service  for  sample 
accountability,  we  could  not  have  made  an  accurate  statistical  analysis 
of  survey  response  rates. 

By  using  military  channels  for  survey  administration  and  imposing 
stringent  requirements  for  survey  accountability  on  military  adminis¬ 
trative  units,  we  obtained  higher  response  rates  than  were  reported  in 
past  DoD  surveys.  The  most  serious  disadvantages  of  this  administrative 
model  were  our  lack  of  control  over  the  actual  data  collection  at  the 
unit  level  and  our  inability  to  make  direct  personal  contact  with  sample 
members.  While  the  use  of  military  channels  contributea  to  improving 
overall  response  rates,  the  survey  was  more  difficult  to  manage  than  most 
traditional  civilian  research  efforts  because  the  actual  day-to-day  data 
collection  activities  were  in  the  hands  of  service  administrators  who 
were  not  directly  accountable  to  Rand.  Poor  execution  of  the  survey  by 
service  administrators  could  obviously  have  negatively  affected  survey 
response  rates,  furthermore,  efforts  to  follow  up  with  nonrespondents 
had  to  be  coordinated  through  the  military  command  and  authority  struc¬ 
ture.  These  procedures  required  close  cooperation  with  our  service 
contacts  who  had  oversight  responsibility  for  the  entire  survey  effort. 
Such  cooperation  ensured  more  effective  follow-up  with  service  adminis¬ 
trators  who  were,  in  turn,  responsible  for  encouraging  sample  members  to 
participate  in  the  survey. 

OTHER  INDICATORS  OF  SURVEY  SUCCESS 

Measures  of  the  success  of  the  survey  administration  model — in 
addition  to  the  overall  response  rates — include  the  unit  participation 
rates  and  the  adherence  to  (1)  the  questionnaire  return  mode,  (2) 
sample  accountability  requirements,  and  (3)  the  survey  administration 
schedule . 
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Unit  Participation  hates 

About  90  percent  of  all  sampled  units  participated  in  the  survey  by 
returning  all  or  some  of  the  data  collection  forms.  To  encourage  this 
high  unit  response  rate,  the  schedule  for  the  data  collection  had  to  be 
considerably  relaxed. 

Questionnaire  Return  Mode 

A  review  of  DoD  prior  experience  in  surveying  Reserve  force 
personnel  suggested  that  mailing  surveys  directly  to  reservists  and 
relying  on  them  to  return  questionnaires  leads  to  extremely  low  response 
rates.  To  improve  survey  response  rates  and  sample  accountability,  the 
survey  was  administered  through  the  unit.  The  commanding  officer  of  each 
selected  unit,  serving  as  the  survey  representative,  was  responsible  for 
distributing  and  collecting  questionnaires  from  all  unit  members.  Each 
unit  was  then  to  return  all  completed  questionnaires  and  sample 
accountability  materials  to  DM DC  in  a  single  shipment.  DMDC  reported 
that  most  units  complied.  Only  200  completed  Enlisted  Personnel 
Surveys ,  representing  about  1  percent  of  all  returned  questionnaires, 
were  returned  by  respondents  themselves  instead  of  through  the  designated 
unit  contact. 

Several  features  of  the  survey  administration  model  contributed  to 
the  overall  success  of  the  questionnaire  distribution  and  return 
procedures,  including  (1)  the  request  that  commanding  officers  assume 
responsibility  for  unit-based  survey  administration,  (2)  the  use  of 
military  time  (monthly  drill  assemblies)  for  survey  administration,  and 
(3)  the  provision  of  sample  units  with  all  of  the  materials  needed  for 
distributing  and  returning  surveys  (including  privacy  envelopes  for 
individual  questionnaires,  return  mailing  boxes,  and  franked  return 
mailing  labels)  to  minimize  the  administrative  burden. 

Sample  Accountability 

As  part  of  the  fieldwork,  sample  units  were  asked  to  account  for  the 
survey  participation  of  all  enlisted  personnel  assigned  to  the  unit  at 
the  time  of  the  survey  administration.  Each  unit  was  sent  a  roster 
containing  the  names  of  the  enlisted  members  of  the  unit  as  shown  on  the 
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Reserve  Components  Personnel  Data  System  at  the  time  of  the  sample 
selection.  The  unit  commander  was  responsible  for  updating  the  roster  to 
reflect  actual  unit  membership  at  the  time  of  the  survey  administration 
and  to  indicate  which  members  participated  in  the  survey.  For  members 
who  did  not  participate,  the  unit  commander  entered  the  reason  on  the 
roster  (for  example,  "absent  frum  drills,"  "on  initial  active  duty 
training /advanced  training,"  or  "refused"). 

Rand  fieldwork  data  show  that  about  bO  percent  of  all  sampled  units 
returned  the  roster  to  DHDC  together  with  completed  surveys,  including 
approximately  88  percent  of  all  sampled  Army  Reserve  units  and  about  77 
percent  of  all  Army  National  Guard  units. 

Survey  Administration  Schedule 

To  obtain  the  highest  possible  survey  response  it  was  essential  that 
we  select  a  survey  period  during  which  most  personnel  would  be  present  in 
their  units  and  could  be  freed  from  their  normal  duties  to  participate  in 
the  survey.  We  determined,  in  consultation  with  the  Department  of 
Defense,  that  survey  administration  from  December  through  January  would 
be  convenient  for  most  units  and  consistent  with  our  analytic  require¬ 
ments.  Fewer  unit  activities  are  generally  planned  for  the  Christmas 
holiday,  and  we  felt  that  most  personnel  would  be  available  to  parti¬ 
cipate  in  the  survey.  Furthermore,  evidence  suggested  that  unit  drill 
attendance  among  enlisted  personnel  might  even  be  higher  during  the 
holiday  season,  when  the  extra  reserve  income  would  be  particularly 
needed.  This  schedule  would  not  interfere  with  major  unit  operational 
requirements,  such  as  field  maneuvers  and  summer  training. 

We  estimated  that  sampled  units  would  need  at  least  two  months  to 
complete  the  unit-based  survey  administration,  excluding  the  time 
required  to  return  completed  questionnaires  through  the  U.S.  mail  system. 
We  wanted  the  unit  commander  to  make  every  reasonable  effort  to  ensure 
that  all  enlisted  unit  members  had  an  opportunity  to  participate  in  the 
survey.  Therefore,  we  requested  that  units  start  the  survey  adminis¬ 
tration  in  December  and  continue  it  through  January  I960  (and  longer  if 
necessary)  to  complete  the  surveys. 

To  enable  us  to  verify  the  actual  month  the  surveys  were  admin¬ 
istered,  we  asked  each  respondent  to  record  on  the  questionnaire  the  date 


the  survey  was  completed.  Previous  discussions  of  questionnaire  returns 
for  this  study  have  focused  on  when  completed  surveys  were  received  in 
DMDC;  however,  we  wanted  to  know  also  when  the  surveys  were  actually 
administered  at  the  unit  level.  Delays  in  the  return  of  completed 
surveys  may  have  resulted  from  either  delays  in  administering  or  return¬ 
ing  the  surveys  or  delays  in  the  U.S.  mail. 

Survey  data  show  that  roughly  85  percent  of  all  returned  surveys 
were  completed  in  December  1979  and  January  1980.  Although  these 
surveys  were  completed  on  schedule,  we  did  not  actually  receive  all  of 
them  until  the  end  of  May  1980.  Units  had  been  instructed  to  return 
completed  surveys  in  one  shipment,  immediately  following  the 
completion  of  the  survey  administration.  We  expected  therefore  to 
receive  completed  surveys  within  four  weeks  from  the  date  materials  were 
shipped  by  sample  units.  Thus,  we  had  projected  that  all  surveys 
completed  in  December  and  January  would  be  received  by  the  end  of 
February  1980;  however,  only  65  percent  of  all  completed  surveys  were 
received  within  this  period. 

There  are  two  possible  explanations  for  the  lags  in  receiving  com¬ 
pleted  materials.  First,  units  may  not  have  mailed  completed  question¬ 
naires  immediately  after  the  survey  was  administered.  In  these  instances, 
the  return  of  materials  may  have  been  delayed  for  a  month  (or  longer), 
until  the  unit's  next  scheduled  monthly  meeting.  We  did  not  expect  such 
delays,  because  the  unit  technician  or  clerk — the  only  person  present  in 
the  unit  on  a  full-time  basis — was  usually  responsible  for  returning 
surveys.  Second,  the  time  required  for  survey  boxes  to  be  returned 
through  the  U.S.  mail  may  have  been  longer  than  we  had  projected.  To 
facilitate  the  questionnaire  return,  we  had  provided  units  with  mailing 
boxes  and  DoD  franked  labels  for  shipping  surveys.  In  either  case,  we  do 
not  know  the  precise  dates  on  which  the  boxes  were  mailed.* 

The  survey  data  were  also  checked  to  see  whether  the  survey  adminis¬ 
tration  was  conducted  during  a  single  month  or  spread  over  several 


•To  guard  against  holiday  season  postal  delivery  problems,  we  asked 
sample  units  to  return  all  packages  by  U.S.  certified  mail,  which 
provides  proof  of  mailing  and  a  receipt.  The  receipt  can  then  be  used, 
if  needed, "TC  trace  lost  pa'Cka'gSS.  MOTt  units,  however,  did  not  certify 
their  returned  boxes  of  surveys;  instead,  they  used  the  mailing  boxes  and 
postage  paid  labels  provided  and  returned  them  by  first-class  mail. 


months  Aporoximately  90  percent  of  all  returned  surveys  were  completed 
during  a  single  month,  with  the  highest  participation  rates  in  December, 
January,  February,  and  April.  In  a  few  instances,  units  completed  the 
questionnaires  over  a  two-  to  three-month  period,  sometimes  in  consecu¬ 
tive  months  and  sometimes  skipping  a  month.  These  data  suggest  that  any 
unit  member  who  was  absent  from  the  reserve  meeting  during  the  month  the 
surveys  were  administered  probably  did  not  have  an  opportunity  to  parti¬ 
cipate  in  the  survey.  Apparently,  most  units  did  not  schedule  follow-up 
interviewing  sessions  for  persons  not  present  during  the  initial  survey 
administration.  We  had  requested  that  units  complete  the  survey  adminis¬ 
tration  during  a  month  when  the  entire  unit  would  be  present  and 
available  to  participate  in  the  survey.  Since  it  was  unlikely  that  100 
percent  unit  attendance  would  have  been  achieved  during  any  single 
monthly  reserve  meeting,  the  unit  commander  used  his  discretion  in 
selecting  the  survey  month  which  he  felt  was  most  convenient  for  his 
unit.  His  decision  may  or  may  not  have  taken  into  account  any  monthly 
variability  in  unit  attendance  rates  which  could  have  had  a  negative 
effect  on  unit  survey  response  rates. 


V.  CONCLUSIONS 


The  field  procedures  developed  for  the  1979  Reserve  Force  Studies 
Surveys  effectively  increased  the  response  levels  well  beyond  those 
obtained  in  past  data  collection  efforts  with  Reserve  Force  personnel. 

We  attribute  this  improvement  to  the  repeated  follow-up  of  sample 
members.  We  established  quality  control  mechanisms  to  spot  problems  in 
the  s”rvey  administration  as  early  as  possible  so  that  appropriate  action 
could  be  taken  to  obtain  the  highest  possible  survey  response.  Advance 
letters  (from  Rand  and  high-ranking  military  officials),  reminder 
letters,  follow-up  packages  containing  additional  copies  of  the 
questionnaires,  and  telephone  calls  all  contributed  to  raising  the 
response  levels. 

The  success  of  the  administrative  procedures  for  this  survey  leads 
us  to  conclude  that  surveys  of  Re.'-rve  Force  personnel  can  be  success¬ 
fully  designed  and  implemented,  if  the  required  technical  and  admin¬ 
istrative  support  is  available.  In  the  opinion  of  the  Rand  staff, 
the  success  of  this  data  collection,  as  well  as  future  surveys  of  Reserve 
and  other  military  populations,  rests  on  the  following  elements: 

o  Visible  support  from  top-level  military  officials. 

Without  a  serious  commitment  of  resources  by  the  Office  of  the 
Secretary  of  Defense  and  the  Services,  primary  data  collection  from 
military  personnel  cannot  be  successfully  implemented.  The  technical  and 
administrative  support  of  the  Office  of  the  Deputy  Assistant  Secretary  of 
Defense  (Reserve  Affairs),  the  National  Guard  Bureau,  and  the  Office  of 
the  Chief,  U.S.  Army  Reserve,  for  the  1979  Reserve  Force  Studies  Surveys 
was  a  critical  factor  in  the  success  of  the  field  procedures.  High- 
ranking  military  officials  fully  supported  the  data  collection  by 
(1)  providing  administrative  resources  for  the  survey  and  sample 
implementation  through  coordination  of  requirements,  arranging  for 
pretests,  resolving  sampling  problems,  and  assisting  in  the  monitoring  of 
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the  field  administration  and  (2)  endorsing  the  survey  in  writing  and 
sending  advance  letters  and  notices  to  sampled  units  and  their  organiza¬ 
tional  superiors  to  encourage  survey  participation. 

o  Use  of  military  channels  for  data  collection. 

Service-specific  administrative  channels  should  be  utilized  in  the 
data  collection  to  improve  response  rates  and  sample  accountability. 
Requesting  that  the  commanding  officers  of  each  sampled  unit  assume  the 
responsibility  for  the  unit-based  survey  administration  and  return  of 
completed  surveys  was  a  key  factor  in  obtaining  unit-level  support  of  the 
data  collection. 

o  Use  of  military  time  for  survey  participation. 

The  information  to  be  collected  in  the  survey  should  be  of  suffi¬ 
cient  importance  to  military  officials  that  duty  time  Ce.g.,  regular 
drill  assemblies)  will  be  authorized  for  group  or  individual  administra¬ 
tion.  Hie  use  of  duty  time  serves  to  legitimize  the  survey  effort  and 
encourage  sample  members  to  participate. 

o  Well-defined  management  structure  for  the  survey  design  and 
administration. 

The  design  and  implementation  of  a  survey  involves  many  different 
tasks,  including  initial  planning,  sample  design,  sample  selection, 
questionnaire  preparation,  pretesting,  data  collection,  data  processing, 
and  report  preparation.  A  well-designed  survey  needs  an  adequate  number 
and  mix  of  staff  members  who  have  the  technical  and  administrative  skills 
to  assume  responsibility  for  various  aspects  of  the  survey  effort.  In 
addition  to  the  administrative  and  technical  support  provided  by  the 
Office  of  the  Deputy  Assistant  Secretary  of  Defense  (Reserve  Affairs), 
the  Army  Reserve  components,  and  the  Defense  Manpower  Data  Center,  the 
approximate  level  of  support  provided  by  Rand  analytic  and  survey  staff 
was  2.5  person-years .  An  essential  part  of  planning  any  data-collection 
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effort  is  understanding  the  sequence  of  steps  involved  in  the  survey  and 
the  technical  and  administrative  details. 

o  Extensive  questionnaire  pretesting. 

Pretesting  the  questionnaire  and  field  procedures  is  the  only  way  to 
find  out  if  everything  works  properly,  especially  if  a  survey  employs  a 
new  procedure  or  new  questions.  A  well-designed  survey  should  include 
provisions  for  at  least  one  pretest,  designed  as  a  small-scale  pilot 
study  to  test  the  feasibility  of  the  intended  field  procedures  and  to 
perfect  the  question  concepts  and  wording  prior  to  the  actual  field 
administration.  Moreover,  second  pretests  are  desirable  to  ensure  that 
the  problematic  questions  have  been  corrected. 

o  Detailed  administrative  instructions. 

Written  procedures  and  administrative  materials  are  needed  to  ensure 
uniform  administration  of  the  survey  by  all  sampled  units  and  to  minimize 
the  administrative  burden  on  those  units.  Special  attention  should  be 
given  to  the  length  and  format  of  administrative  instructions,  as  unduly 
long  or  complex  procedures  are  apt  to  induce  poor  implementation.  To 
facilitate  the  field  implementation  and  encourage  survey  participation, 
sampled  units  should  be  given  all  of  the  materials  necessary  for 
returning  completed  materials,  including  privacy  envelopes  for  individual 
questionnaires,  return  mailing  boxes,  and  franked  mailing  labels. 

o  Interactive  field  monitoring  with  follow-up  of  nonparticipants. 

Failure  to  follow  up  nonrespondents  in  the  1979  Reserve  Force 
Studies  Surveys  would  have  resulted  in  a  much  lower  survey  response 
rate  and  seriously  limited  the  analytic  value  of  the  collected  data.  Our 
survey  experience  indicated  that  with  repeated  contact  by  Rand  and  the 
Army  Reserve  components,  many  units  that  initially  had  not  been  inclined 
to  participate  were  encouraged  to  return  completed  surveys.  Repeated 
follow-up  by  both  Rand  and  military  officials  resulted  in  about  SO 
percent  of  the  sampled  units  participating  in  the  survey. 
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Although  the  fieldwork  for  the  1979  Reserve  Force  Studies  Surveys 
went  extremely  well  for  the  most  part,  in  monitoring  the  field  adminis¬ 
tration  we  identified  several  problems  in  the  distribution  and  return  of 
questionnaires.  Subsequent  data  collection  efforts  with  Reserve  Force 
personnel  might  be  able  to  avoid  (or  at  least  minimize)  these  problems  by 
handling  some  aspects  of  the  survey  operations  differently. 

The  delivery  of  materials  to  sampled  units  was  delayed  in  some  cases 
because  of  unit  address  problems.  Current  addresses  obtained  from  the 
Army  Reserve  components  were  sometimes  incomplete  and/or  out-of-date. 
Early  in  the  survey  implementation  phase,  we  found  that  about  8  percent 
of  our  sample  list  had  incomplete  addresses.  Later,  we  encountered  two 
other  major  address  problems.  First,  the  use  of  addresses  containing  a 
post  office  box  number  rather  than  a  street  mailing  address  delayed  the 
delivery  of  survey  boxes.  Second,  the  use  of  addresses  that  failed  to 
identify  the  unit  properly  resulted  in  undeliverable  boxes.  The  latter 
problem  occurred  more  frequently  at  the  larger  military  bases,  where  many 
active  and  reserve  units  are  headquartered  and  exact  unit  identification 
is  needed  for  mailroom  distribution.  Since  the  mailing  labels  did  not 
contain  the  name  of  the  unit  contact,  but  were  addressed  to  the 
"Commander  Officer  of  Unit  X,"  delivery  to  the  correct  unit  depended  on 
the  accuracy  and  completeness  of  the  unit  name  and  address.  By  delaying 
the  delivery  of  survey  materials  to  sample  units,  these  address  problems 
also  delayed  the  start  of  the  survey  administration.  To  avoid  these 
problems  in  the  future,  survey  mailing  labels  should  contain  (1)  the  name 
of  unit  contact  to  facilitate  distribution  of  materials,  (2)  the  service, 
e.g..  Army  National  Guard  or  Army  Reserve,  (3)  the  complete  unit 
designation,  including  numeric  identification  (e.g.,  210th  Transportation 
Company),  and  (4)  complete  street  mailing  address  rather  than  post  office 
box  number. 

Several  aspects  of  the  shipping  procedures  for  the  distribution  and 
return  of  questionnaires  could  have  been  nandled  more  efficiently. 
Although  advance  notification  was  sent  to  sampled  units  alerting  them  of 
the  survey  requirements  and  the  approximate  schedule,  they  did  not  know 
when  the  survey  materials  would  be  delivered.  As  a  result,  units  that 
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did  not  receive  the  original  survey  shipments  did  not  report  that 
materials  had  not  been  received.  We  had  to  rely  on  the  delivery  records 
of  the  shipper — the  United  Parcel  Service — to  identify  potential 
nonreceipt  problems.  It  might  be  useful  to  send  a  shipment  verification 
letter  to  all  sampled  units  at  the  same  time  the  surveys  are  mailed, 
telling  them  the  expected  date  of  the  shipment  delivery  and  requesting 
that  they  call  a  designated  point-of-contact  immediately  if  the  surveys 
are  not  received  by  that  time. 

Our  field  experience  in  handling  the  returned  questionnaires  for 
this  study  suggests  that  subsequent  surveys  of  Reserve  Force  personnel 
should  adopt  procedures  to  accelerate  the  return  of  questionnaires.  We 
found  that  many  units  did  not  return  the  surveys  as  soon  as  they  had 
completed  them.  Our  field  monitoring  procedures  had  indicated  early  in 
the  fieldwork  that  the  survey  materials  had  not  been  returned,  and  we  had 
assumed  that  the  delays  were  due  to  the  units'  slowness  in  administering 
the  surveys;  only  later  did  we  learn  that  many  of  the  units  had  completed 
the  surveys  as  scheduled  but  had  failed  to  return  them  immediately.  Our 
analysis  of  completed  surveys  shows  that  about  85  percent  had  been 
completed  during  the  original  two-month  survey  period  (December  1979  and 
January  1580);  the  remaining  questionnaires  (15  percent)  were  completed 
during  the  following  five  months  (February  through  June  I960).  Thus,  the 
extension  of  the  survey  period  to  seven  months  was  necessitated  more  by 
the  problem  of  getting  most  units  to  return  materials  expeditiously  than 
by  the  problem  of  getting  them  to  complete  the  surveys. 

Future  data  collection  efforts  should  probably  use  special  mailing 
procedures,  such  as  certified  or  special  delivery  letters  and  mailgrams, 
or  telephone  calls  to  encourage  survey  participants  to  return  survey 
materials  promptly.  Future  surveys  might  also  ask  survey  administrators 
responsible  for  the  data  collection  to  return  a  postcard  indicating  that 
they  mailed  questionnaires  under  separate  cover,  on  a  given  date,  by  a 
given  shipment  method.  The  survey  team  could  expect  to  receive  such 
postcards  much  sooner  than  the  actual  shipment  of  surveys,  enabling  them 
to  check  response  rates  more  rapidly  than  was  possible  for  the  present 
survey.  The  failure  of  units  to  return  postcards  would  signal  potential 
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field  problems  early  in  the  field  period  and  thus  facilitate  efforts  to 
follow  up  nonrespondents. 

Researchers  involved  in  subsequent  data  collection  from  Reserve  and 
other  military  populations  should  consider  assessing  the  effectiveness 
of  using  alternative  procedures  and  approaches  for  increasing  response 
levels.  Future  methodological  work  should  build  on  our  experience  in 
conducting  the  1979  Reserve  Force  Studies  Surveys  to  develop  a 
systematic  body  of  knowledge  regarding  the  factors  that  produce  the 
highest  possible  survey  response  among  military  personnel. 
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Appendix  A 

STEPS  REQUIRED  TO  CONDUCT  THE  1979  RESERVE  FORCE  STUDIES  SURVEYS 


This  appendix  list  in  detail  the  eight  tasks — preliminary  survey 
planning,  questionnaire  preparation,  sample  design  and  implementation, 
survey  administration  design,  data  collection  and  monitoring,  survey 
returns  and  editing,  data  processing,  and  survey  documentation — involved 
in  the  design  and  execution  of  the  1979  Reserve  Force  Studies  Surveys, 
together  with  the  approximate  amount  of  time  required  to  complete  each 
activity.  Immediately  following  the  description  of  each  survey  task  are 
notes  providing  additional  information  on  the  relationships  between 
various  survey  processes — information  that  may  affect  staff  and  time 
allocations  and  alert  the  user  to  procedural  details  that  should  be 
considered  in  subsequent  data  collections  from  Reserve  Force  personnel. 


TASK  1:  PRELIMINARY  SURVEY  PLANNING 


1,1  Initial  concept  development  for  overall  study  design — 8  weeks 
o  Identify  information  needs  for  the  study 
o  Specify  research  objectives 

o  Review  preliminary  ideas  with  client/Rand  staff 

o  Develop  initial  research  proposal 
1.2  Preparation  of  survey  planning  documents  by  analytic  and 
survey  staff — 3  weeks 

o  Issue  paper  on  research  issues  in  survey,  sample  and 
and  administration  design 

o  Survey  design  paper  describing  major  survey  activities 
project  milestones,  planning  schedules,  and  delineating 
organizational/staff  responsibility  for  various  survey 
processes 
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1.3  Conduct  in-house  Rand  seminar  for  manpower  staff  knowledgable 
about  Reserve  force  policy  issues — 1  day.  Primary  purposes 
of  seminar  are  to: 

o  Discuss  survey  planning  documents  (see  Task  1.2) 

o  Identify  staff  interest  in  participating  in  the  survey 

design 

1.4  Initiate  planning  discussions  with  staff  at  ODASD  (Reserve 
Affairs)  and  Army  Reserve  components  regarding  survey 
requirements — 1  week 

o  Conduct  planning  meeting  at  DoD 
o  Provide  background  data  on  proposed  survey  plans 
o  Solicit  input  from  client  on  overall  study  design 

1.5  Obtain  primary  points-of-contact  (PPOCs)  from  Army  Reserve 
components  to  assist  in  survey  implementation — 1  week 

1.6  Obtain  preliminary  data  requirements  (e.g.,  variable  lists 
and  suggested  question  items)  from  Rand  staff  participating 
in  survey  design — 4  weeks 

1.7  Conduct  preliminary  site  visits  to  Reserve  Force  units  at 
regular  drill  assemblies  and  at  annual  training  sites — 4  weeks 
o  Survey  and  analytic  teams  visit  selected  units 

o  Conduct  informal  interviews  with  Reserve  Force  personnel 
o  Obtain  information  about  the  structure  and  organization 

of  units  and  other  data  needed  to  design  the  survey  instruments 

1.8  Refine/finalize  data  requirements  on  the  basis  of  preliminary 
interviews  and  further  input  from  Rand  analysts — 4  weeks 

1.9  Analysts  provide  survey  staff  with  list  of  variables  so 
that  questionnaire  development  can  begin 

NOTES  FOR  TASK  1 

1.  Most  activities  under  this  task  were  sequential  and  depended  on 
the  completion  of  a  prior  step:  For  example,  planning  discussions  with 
the  client  did  not  begin  (see  Task  1.4)  until  after  tne  proposed  survey 
plans  had  been  documented  and  discussed  with  manpower  analysts  at  Rand 
(see  Tasks  1. 1-1.3).  The  exceptions  were  Task  1.7  (preliminary  site 
visits)  and  Task  1.8  (finalization  of  data  requirements),  which 
overlapped  and  were  completed  simultaneously.  The  end  result  of  these 
two  tasks  was  Task  1.9 — the  production  of  a  list  of  recommended  variables 
for  the  survey  instruments. 


2.  The  approximate  time  period  ior  completing  these  planning 
activities  was  roughly  five  months,  from  January  1979  and  through  to  May 
1979. 

3.  Work,  could  not  begin  on  survey  tasks  2  through  8  until  after  all 
of  the  preliminary  survey  planning  activities  were  completed. 

TASK  2:  QUESTIONNAIRE  PREPARATION 

2.1  Review  previous  data  collection  efforts  with  Reserve  Force 
population,  related  DoD  surveys  and  related  civilian 
surveys — 2  weeks 

2.2  Draft  questionnaires  using  design  specifications  provided 
by  Rand  analysts — 4  weeks 

2.3  Coordinate  and  review  draft  questionnaires  with  staff  at 
Rand/ODASD  (Reserve  Affairs)  and  Army  Reserve — 5  weeks 

2.4  Revise  questionnaires,  as  required;  prepare  for  pretesting 
with  Reserve  Force  personnel — 2  weeks 

2.5  Develop  and  coordinate  pretest  requirements  with  ODASD 
(Reserve  Affairs)  and  the  Army  Reserve  Components — 4  weeks 

2.6  Conduct  large  scale  pretest  at  selected  Annual  Training 
Bases  and  analyze  results;  conduct  smaller  pretests, 

as  necessary,  to  test  final  four  questionnaires — 4  to  5  weeks 

2.7  Final  questionnaire  revisions  and  production  of  printer- 
ready  Cv  jies;  circulate  for  final  review  by  Rand  and  ODASD 
(Reserve  Affairs) — 4  weeks 

o  Enlisted  Forms  1  and  2  were  designed  as  optical  scan 
forms 

o  Unit  and  Commander  Forms  3  and  4  were  designed  as  key¬ 
punch  forms 

2.8  Obtain  survey  clearance  from  ODASD  (Reserve  Affairs)  (e.g., 

RCS  approval);  prepare  supporting  justii ication ;  add  RCS 
number  to  survey  instruments  and  procedures — 3  to  4  weeks 

2.9  Provide  printing  contractor  with  final  printer-ready 
questionnaires  and  printing  specifications  (i.e.,  formats, 
color  selection,  printing  quantities,  etc.)  —  1  day 
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2.10  Contractor  submits  preliminary  mock-ups  of  questionnaires 
as  well  as  printer-ready  proof  copies  for  Rand  review  and 
approval — 3  to  5  days 

o  Rana  approves  final  questionnaire  layout  and  format 
o  Contractor  make  questionnaire  modifications  as  specified 
by  Rand 

2.11  Print  final  questionnaires  for  self-administration 
o  Enlisted  Forms  1  and  2 — 6  weeks 

o  Unit  and  Commander  Forms  3  and  4 — 2  weeks 


NOTES  FOR  TASK  2 

1.  Most  of  the  activities  under  this  section,  like  those  in  Task  1, 
were  sequential  and  conducted  in  the  order  listed.  The  two  exceptions 
were  Task  2.5  (coordinating  pretest  requirements)  and  Task  2.8  (obtaining 
survey  clearance).  We  reviewed  the  pretest  requirements  (Task  2.5)  with 
the  DoD  client  and  the  Army  Reserve  components  at  the  same  time  the  draft 
questionnaires  were  being  discussed  (Task  2.3).  We  obtained  survey 
clearance  for  the  data  collection  (Task  2.8)  as  soon  as  the  question¬ 
naires  and  operational  plans  for  the  study  were  in  near-final  form,  that 
is,  soon  after  the  questionnaire  pretest  was  completed  and  analyzed  (Task 
2.6) . 


2.  Approximately  four  and  one-half  months  were  required  to  design, 
pretest,  revise,  and  finalize  the  four  separate  data  collection  instru¬ 
ments  developed  for  this  study.  An  additional  six  weeks  were  added  to 
the  questionnaire  preparation  stage  to  cover  the  time  needed  for  format¬ 
ting  and  printing  the  questionnaires  designed  for  optical  scan  process¬ 
ing.  (The  Unit  and  Commander  questionnaires  were  printed  during  a  two- 
week  period  which  overlapped  with  the  printing  schedule  for  the  other 
two  survey  instruments.) 

3.  The  fieldwork  schedule  for  the  1979  Reserve  Force  Studies 
Surveys  was  driven  in  large  part  by  the  tlwe  requirements  Tor  producing 
ilnal  questionnaires.  Most  of  the  remaining  tasks  took  considerably  less 
time  and  could  be  completed  at  the  same  time  as  Task  2.0. 


TASK  3:  SAMPLE  DESIGN  AND  IMPLEMENTATION 


3.1  Develop  preliminary  sample  design/selection  plans — 3  weeks 

3.2  Initiate  planning  discussions  with  staff  at  Rand/ODASD 
(Reserve  Affairs)  and  the  Army  Reserve  components — 1  week 

3.3  Obtain  primary  point-of-contact  for  sampling  activities 
from  Army  Reserve  components — 1  week 


56 


3.**  Obtain  DMDC  point-of-contact  to  assist  in  sampling 

activities  and  preparation  of  sample  materials  needed  for 
survey  administration — 1  week 

3.5  Provide  DMDC  with  preliminary  data  processing  specifications 
for  sampling  activities — 1  week 

3.6  Select  sample  of  units — b  weeks 

3.7  Obtain  most  current  mailing  addresses  for  sample  units  and 
pay-grade  distribution  of  enlisted  personnel  (from  DMDC  and 
Army  Reserve  components) — 2  weeks 

3.6  Resolve  sample  problems — 2  to  3  weeks 

o  Identify  sampled  units  which  were  reorganized  or  inactivated 
since  sample  selection;  supplement  the  sample  as  needed  to 
replace  these  units 
o  Identify  split  units 

o  Resolve  any  remaining  sample  related  problems 

3-9  DMDC  preparation  of  sample  files  and  unit  rosters  containing 

unit  mailing  addresses  and  names  of  all  known  El  to  E9  personnel 
assigned  to  each  sampled  unit — 12  weeks 

3.10  DMDC  mail  sample  tapes  and  unit  rosters  to  mailing  contractor 
responsible  for  the  assembly  and  distribution  of  survey 
materials — 1  week 


NOTES  FOR  TASK  3 

1.  All  sampling  activities  overlapped  with  those  tasks  completed 
during  the  six  month  questionnaire  design  phase  (Task  2.0).  This  means 
that  the  sample  design  and  implementation  was  being  carried  out  at  the 
same  time  the  questionnaires  were  being  drafted,  pretested,  revised,  and 
put  into  final  form.  Furthermore,  the  analysts  and  survey  staff 
responsible  for  the  sample  design  and  implementation  were  also 
responsible  for  developing  and  finalizing  the  data  collection 
instruments. 

2.  Task  3.1  (develop  preliminary  sampling  plans)  was  completed  as 
part  of  the  initial  survey  planning  stage.  These  plans  were  incorporated 
into  the  survey  planning  documents  discussed  under  Task  1.2. 

3.  Many  of  the  sampling  activities  could  have  been  carried  out  more 
or  less  independently  of  other  survey  tasks.  For  example,  the  selection 
of  the  sample  units  could  have  taken  place  at  the  same  time  the  question¬ 
naires  were  being  pretested  ana  evaluated.  However,  the  actual  schedule 
for  completing  individual  sampling  activities  was  a  function  of  available 
staff  and  resources  for  conducting  concurrent  survey  processes.  We  found 
it  necessary  to  delay  some  sampling  activities  until  the  available  staff 
could  be  freed  from  other,  more  critical  tasks.  On  the  whole,  we  found 
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that  we  were  able  to  keep  activities  in  Tasks  2  and  3  moving  on  a 
parallel  track  with  minimal  problems  so  that  both  the  final  question¬ 
naire  and  sample  would  be  ready  at  the  same  time. 

3.  Because  of  the  overlap  of  Tasks  2  and  3.  it  is  difficult  to 
estimate  the  total  time  required  for  sampling  activities  alone. 
Approximate  tITrre- estimates  for  individual  sampling  activities  are  listed 
in  the  outline. 

TASK  4:  SURVEY  ADMINISTRATION  DESIGN 

4.1  Identify  previous  data  collection  efforts  with  Reserve  Force 
population/related  DoD  surveys;  develop  preliminary  survey 
administration  plans — 2  to  3  weeks 

4.2  Initiate  planning  discussions  with  staff  at  Rand,  .^ASD  (Reserve 
Affairs) ,  and  the  Army  Reserve  components — 1  week 

4.3  Obtain  primary  points-of-contact  (PPOCs)  from  Army  Reserve 
components;  identify  the  technical  and  administrative  support 
needed  from  service  PPOCs — 1  week 

4.4  Prepare  and  coordinate  authority  letters  and  advance  notices 
from  high-level  military  leaders  to  sampled  units  and  their 
organizational  superiors — 6  to  7  weeks 

4.5  Rand  sends  advance  letters  to  unit  commanders  of  all  sample 
units  and  their  organizational  superiors — 2  weeks 

4.6  DM DC  procured  administrative  materials  needed  for  survey  (privacy 
envelopes,  acknowledgment  postcards,  DoD  franked  labels  for  return 
mailing,  etc.) — 4  weeks 

4.7  Develop  and  finalize  administrative  and  sample  accountability 
procedures/ forms  after  appropriate  review  with  ODhSD  (Reserve 
Affairs)  and  the  Army  Reserve  components — 4  weeks 

4.8  Provide  mailing  contractor  with  final  specifications  for  produc¬ 
ing  and  distributing  administrative  materials  to  sampled  units 

NOTES  FOR  TASK  4 

1.  The  initial  design  of  the  survey  administration  procedures  was 
completed  during  the  preliminary  survey  planning  phase  (Task  1.2).  This 
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enabled  us  to  obtain  early  agreement  from  ODASD  (Reserve  Affairs)  and  the 
Army  Reserve  components  to  fully  support  the  data  collection  require¬ 
ments. 

2.  Authorization  letters  from  high  ranking  military  officers  in  the 
ODASD  (Reserve  Affairs)  and  the  Army  Reserve  components  to  sampled  units 
and  their  organizational  superiors  were  needed  to  obtain  unit-level 
support  of  tne  data  collection.  These  letters  specified  the  survey 
administrative  requirements  and  requested  cooperation  from  units  and 
their  immediate  organizational  superiors.  Rand  was  responsible  for 
drafting  the  letters  and  reviewing  them  with  our  service  PPOCs.  Review 
meetings  to  discuss  the  proposed  content  of  the  required  letters  were 
coordinated  through  the  client — the  ODASD  (Reserve  Affairs).  Given 
organizational  differences  between  the  Army  Reserve  components,  a 
separate  set  of  letters  was  developed  for  each  component,  as  shown  below. 

For  the  Army  National  Guard: 

a.  Letter  from  ODASD  (Reserve  Affairs)  to  the  chief.  National  Guard 
Bureau 

b.  Letter  from  chief.  National  Guard  Bureau,  to  adjutant 
general  of  each  state 

c.  Letter  from  state  adjutant  general  to  commanding  officer  of 
each  sampled  unit 

d.  Letter  from  ODASD  (Reserve  Affairs)  to  enlisted  sample  members 
For  the  Army  Reserve: 

a.  Letter  from  ODASD  (Reserve  Affairs)  to  chief.  Army  Reserve 

b.  Letter  from  chief,  Army  Reserve  to  U.S.  Forces  Command  (FORSCOM) 

c.  Letter  from  FORSCOM  to  commanding  officer  of  each  sampled  unit 

d.  Letter  from  ODASD  (Reserve  Affairs)  to  enlisted  sample  members 

3.  Given  the  lead  time  required  to  prepare  and  coordinate  the 
various  military  authority  letters,  Rand  initiated  these  activities 
approximately  three  months  before  the  data  collection  was  scheduled  to 
begin.  This  enabled  us  to  send  out  the  advance  notification  to  the 
individual  units  A  to  6  weeks  before  the  scheduled  field  period.  We  feel 
that  this  was  the  minimum  lead  time  needed  by  sampled  units  to  plan  for 
the  data  collection. 

4.  Most  of  the  activities  under  this  task  (4.4  through  4.8)  were 
completed  by  the  survey  staff  as  soon  as  the  questionnaire  was  finalized 
and  ready  for  final  production  and  the  final  sample  list  had  been 
prepared.  These  activities  could  have  been  completed  earlier,  if  staff 
could  have  been  freed  from  Tasks  2  and  3;  however,  we  felt  that  the 
activities  should  be  delayed  until  the  questionnaire  and  sample  had  been 
finalized.  In  a  few  instances,  last  minute  changes  in  the  questionnaire 
and  sampling  strategy  resulted  in  changes  to  the  administrative  require¬ 
ments  . 


TASK  5:  DATA  COLLECTION  AND  MONITORING 


5.1  Obtain  unit  points-of-contact  and  PPOCs  for  their  organiza¬ 
tional  superiors;  the  command  contact  for  Guard  units  were 
located  at  the  state  adjutant  general's  office;  in  the 
reserves,  the  command  liaison  was  stationed  at  U.S.  FORSCOM' 
2  to  3  weeks 


5.2  Contractor  mailing  of  questionnaires/ administrative  materials/ 
sample  materials  directly  to  sample  units — 3  days 

5.3  Survey  administration  begins  at  sampled  units 

5.4  Rand  implements  procedures  to  verify  receipt  of  questionnaires 
by  all  units  which  did  not  acknowledge  shipments — 2  weeks 

5.5  Rand  set  up  fieldwork  monitoring  procedures  at  Rand/DMDC — 

2  weeks 

5.6  Develop  follow-up  strategies/procedures  for  nonparticipating 
units  and  units  which  returned  incomplete  shipments  (e.g., 
missing  Unit  and/or  Commander  Forms) ,  after  appropriate 
review  with  service  PPOCs — 2  weeks 

5.7  Identify  units  which  lost/misplaced  initial  survey  materials; 
send  second  shipment,  as  necessary — 1  week 

5.8  Identify  units  which  returned  partial  materials  not  including 
a  Unit  and/or  Commander  Form;  send  Rand  letter  requesting 
completion  of  these  two  forms  and  extending  the  survey 
suspension  date  to  May  31.  I960 — 1  week 

5.9  Review  and  coordinate  follow-up  letters  from  Service  PPOCs 
to  nonparticipating  units  requesting  cooperation  and 
extending  the  survey  suspension  date  to  May  31,  1980 — 2  to  3 
weeks 

5.10  Final  Rand  follow-up  letter  to  nonparticipating  units 
requesting  completion  of  only  one  of  the  survey  forms — the 
unit  form;  suspension  date  extended  to  June  30,  1980 — 1  week 

5.11  Return  mailing  of  completed  questionnaires/sample  materials 
to  DMDC — 7  months 

5.12  Write  regular  survey  progress  reports  for  Rand,  ODASD 
(Reserve  Affairs),  and  the  Army  Reserve  components 

NOTES  FOR  TASK  5 

1.  Activities  under  this  section  were  conducted  during  the  seven- 

month  fieldwork  period  from  November  1979  through  June  1980. 

2.  Tasks  5.4  through  5.10  represent  successive  follow-up  strategies 

implemented  by  Rand  survey  staff  and  the  Army  Reserve  components  to 

encourage  a  high  survey  response. 


3.  We  found  it  necessary  to  extend  the  survey  field  period  to  seven 
months  to  obtain  the  highest  possible  survey  response  from  the  sampled 
units.  The  rate  of  questionnaire  return  was  considerably  slower  than 
expected  because  (1)  roughly  20  percent  of  the  sampled  units  did  not 
complete  the  surveys  during  the  original  two  month  survey  period  (after 
repeated  follow-up,  about  naif  of  these  units  participated  in  the  survey 
between  February  and  June  I960);  (2)  many  units  that  adhered  to  the 
original  survey  schedule  apparently  did  not  return  the  surveys 
immediately  after  completing  them.  We  discovered  in  analyzing  the 
completed  questionnaires  that  most  units  (about  80  percent)  had  actually 
administered  the  surveys  during  the  first  two  months  of  the  field  period; 
however,  we  did  not  receive  all  of  these  questionnaires  until  near  the 
end  of  the  field  period.  Tne  lags  in  receiving  completed  surveys  were 
due  to  either  unit  delays  in  returning  the  questionnaires  or  delays 
caused  by  the  U.S.  Postal  Service's  handling  of  bulk  mail. 

If  procedures  can  be  developed  to  increase  the  rate  of  question¬ 
naire  returns,  subsequent  data  collections  may  not  require  such  a  lengthy 
field  period.  Our  survey  experience  suggests  that  flexibility  in  extend¬ 
ing  the  field  period  was  a  crucial  factor  in  obtaining  high  response 
rates  among  Reserve  Force  personnel. 


TASK  6:  SURVEY  RETURNS  AND  EDITING 


6.1  DMDC  received  returned  questionnaires/ sample  rosters  from 
participating  units — 7  months 

6.2  Rand  developed  procedures/ forms  for  questionnaire  receipt 
and  editing;  trained  DMDC  personnel — 1  to  2  weeks 

6.3  DMDC  implemented  survey  receipt  and  editing  procedures — 6  months 
Major  requirements  were  to 

o  Maintain  records  of  returned  shipments  (date  received  at 
DMDC,  number  and  type  of  forms  received) 
o  Enter  Record  Control  Number  (RCN)  on  each  questionnaire 
to  identify  the  respondent's  sample  unit 
o  Send  to  Rand,  on  a  regular  basis,  document  control  forms 
for  each  unit,  specifying  exact  materials  returned  to 
DMDC 

o  Send  Unit  and  Commander  forms  regularly  to  Rand  for 
prekeypunch  editing/coding 

o  Edit  and  prepare  enlisted  personnel  surveys  (Forms  1  and 
2)  for  shipment  to  optical  scanning  contractor  for 
processing 


61 


6.4  Developed  editing  and  coding  procedures  for  Unit  and  Commander 
Forms  (3  and  4);  train  Rand  editors — 2  weeks 

6.5  Rarid  edited,  coded,  and  prepared  Unit  and  Commander  Forms 
for  keypunching;  sent  forms,  on  a  regular  basis,  to  DMDC 
for  keypunching  and  subsequent  processing — 20  weeks 


NOTES  FGk  TASK  6 

1.  Completed  surveys  for  this  study  were  received  by  DMDC  from 

•  late  December  1979  through  July  1980.  During  WITS'  "period ,  DMDC  provided 
approximately  3.5  full-time  equivalents  (FTEs)  to  handle  the  clerical 
activities  associated  with  receiving  and  editing  of  over  20,000  ques¬ 
tionnaires  from  sampled  units. 

2.  Over  a  five  month  period,  research  assistants  at  Rand  (1.5  FTEs) 
were  responsible  for  editing  and  preparing  the  Unit  and  Commander  Surveys 
for  subsequent  processing  at  DMDC. 


TASK  7:  DATA  PROCESSING 


7.1  Selection  of  data  processing  mode  (Rand/DMDC  staff) — 2  weeks 
o  Enlisted  Forms  1  and  2:  OPSCAN  (Contractor) 

o  Unit  and  Commander:  Keypunch  (DMDC) 

7.2  DMDC  procure  optical  scanning  contractor  to  process  data  for 
Forms  1  and  2;  Rand  provided  DMDC  with  technical  specification  for 
contract — 12  weeks 

7.3  Rand  provided  optical  scan  contractor /DMDC  with  coding  specifi¬ 
cation  for  initial  data  processing — 2  weeks 

o  Forms  1  and  2:  OPSCAN  (Contractor) 

o  Forms  3  and  4:  Keypunch  (DMDC) 

7.4  Initial  batch  processing  of  questionnaires 

o  Forms  1  and  2:  OPSCAN  (Contractor) — 20  weeks 

o  Forms  3  and  4:  Keypunch  (DMDC) — 15  weeks 

7.5  DMDC  data  cleaning  and  reduction — 32  weeks 
Major  activities  include 

o  Coding  verification 
o  Recoding  blank  responses/problem  codes 
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o  Editing  out-of-range  data/ imbedded  blanks/ incomplete 
numeric  responses/other  coding  problems 
o  Performing  preliminary  data  quality  checks 
o  Create  SPSS  system  files/documentation 
7.6  Produce  final,  edited  data  tapes  for  analysis 


NOTES  FOR  TASK  7 

1.  DMDC's  procurement  of  an  optical  scan  contractor  (Task  7.2) 
required  approximately  three  months.  Under  a  special  contractual 
arrangement,  DMDC  obtained  a  single  contractor — the  Intran  Corporation-- 
to  provide  data  processing  support  for  all  surveys  under  DMDC's  jurisdic¬ 
tion  during  that  fiscal  year.  This  special  arrangement  enabled  DMDC  to 
complete  the  subcontractor  negotiations  within  a  shorter  time  than  is 
normally  required.  Subsequent  data  collection  efforts  may  find  it 
necessary  to  allow  up  to  six  months'  lead  time  to  procure  a  data 
processing  contractor  through  the  DoD  contracting  office. 

2.  To  minimize  processing  delays,  the  questionnaires  were  processed 
in  batches  as  materials  were  returned  from  the  field.  Thus,  the  initial 
data  processing  was  spread  over  a  five-month  period. 

3.  Planning  at  DMDC  for  additional  computer  editing  and  data 
cleaning  and  the  production  of  SPSS  system  files  began  in  January  1980, 
as  soon  as  the  initial  data  processing  specifications  were  developed  by 
Rand.  The  DMDC  data  processing  team  consisted  of  a  seni( programmer 
analyst,  assisted  by  1  or  2  junior  programmers  and  a  data  technician  with 
expertise  in  creating  SPSS  system  files.  The  production  of  final  edited 
data  tapes  for  analysis  was  spread  over  an  eight-month  period.  Rand 
survey  staff  worked  closely  with  the  DMDC  team  to  provide  technical 
specifications  for  all  data  processing  activities. 


TASK  8:  SURVEY  DOCUMENTATION 


8.1  Rand  survey  and  analytic  staff  documented  this  survey  in  the  follow¬ 
ing  three  publications: 

o  N-1749-MRAL:  1979  Reserve  Force  Studies  Surveys: 

Survey  Design,  Sample  Design,  and  Administrative 
Procedures 

o  N-1755-MRAL:  1979  Reserve  Force  Studies  Surveys: 

User's  Manual  and  Codebook 

o  N-1750-MRAL:  1979  Reserve  Force  Studies  Surveys: 

Description  and  Evaluation  of  Survey  Procedures 


**  *  : 
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NOTES  FOR  TASK  8 

1.  Decisions  were  made  at  the  start  of  the  project  in  January  1979 
about  the  type  of  survey  documentation  to  be  produced  at  the  end  of  the 
study.  Believing  that  it  was  crucial  to  provide  information  about  how 
the  survey  was  designed  and  executed  and  data  for  assessing  the  field 
experience,  we  wrote  preliminary  drafts  of  the  planned  reports  during  the 
survey  planning  stage.  Throughout  the  survey  design  and  implementation 
phase,  we  maintained  detailed  records  of  procedures  and  our  experiences  in 
monitoring  the  data  collection.  The  preliminary  planning  and  detailed 
record-keeping  greatly  facilitated  the  writing  of  the  final  survey 
reports. 

2.  Since  much  of  the  work  of  producing  these  reports  was  completed 
as  part  of  the  survey  design  and  implementation  (Tasks  1  through  7),  it 
is  difficult  to  estimate  the  total  time  required  for  producing  the 
three  survey  documents.  Once- ST1- of  the  materials  had  been  assembled  and 
analyzed,  it  took  approximately  4  to  6  weeks  to  prepare  each  of  the  final 
reports. 


I 
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Appendix  5 

FOLLOW-UP  LETTERS  AND  NOTICES 

This  appendix  contains  copies  of  the  follow-up  letters  and  notices 
sent  by  Rand  and  the  Army  Reserve  components  to  sampled  units  to  encourage 
survey  participation.  A  summary  of  the  content  and  purpose  of  these 
letters  is  provided  below: 

1.  Letter  dated  1  April  1980,  from  Jennifer  A.  Hawes,  Survey 
Coordinator  at  The  Rand  Corporation,  to  state  adjutants 
general,  SUBJECT:  1979  Reserve  Force  Studies  Surveys:  Status 
Report. 

This  letter  informed  the  state  adjutants  general  of  the  status  of 
the  survey  administration  for  National  Guard  units  as  of  March  1980  and 
requested  their  assistance  in  following  up  units  that  had  not  participated 
in  the  survey.  In  response  to  the  Rand  request,  the  POC  in  the  appropri¬ 
ate  adjutant  general's  office  contacted  nonparticipating  units  (either 
by  phone  or  military  messages) ,  requested  their  cooperation  with  the 
survey,  and  informed  them  that  the  survey  period  had  been  extended  to 
May  31.  1980. 

2.  Letter  dated  2  April  1980,  from  David  Grissmer,  Deputy  Director, 
Military  Manpower,  Mobilization  and  Readiness  Program,  The  Rand 
Corporation,  to  the  commanding  officers  of  sampled  units, 

SUBJECT:  1979  Reserve  Force  Studies  Surveys:  Shipment 
Verification. 

This  follow-up  letter  was  mailed  to  each  unit  (Reserve  and  National 
Guard)  that  returned  completed  enlisted  personnel  surveys,  but  failed 
to  send  a  Unit  and/or  Commander  Form.  Units  were  asked  to  complete  and 
return  the  missing  form(s)  no  later  than  31  May  1980. 
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3.  Letter  dated  6  April  1980,  from  Jennifer  A.  Hawes,  The  Rand 

Corporation,  to  state  adjutants  general,  SUBJECT:  1979  Reserve 
Force  Studies  Surveys:  Suspense  Date  (May  31.  I960). 

This  letter  provided  an  updated  status  report  on  the  survey 
administration  for  National  Guard  units.  Rand  indicated  that  units 
were  notified  that  the  survey  period  had  been  extended  to  31  May  1980 
to  enable  them  to  complete  those  Commander  and/or  Unit  Forms  that  had 
not  been  returned  in  the  shipments  of  completed  surveys. 

U.  Letter  dated  5  June  I960,  from  Jennifer  A.  Hawes,  The  Rand 

Corporation,  to  state  adjutants  general,  SUBJECT:  1979  Reserve 
Force  Studies  Surveys:  Updated  Survey  Report. 

This  final  letter  to  our  Guard  contacts  reported  on  the  status  of 
the  survey  administration  as  of  5  June  I960.  The  letter  identified 
units  that  had  not  returned  any  completed  surveys  and  informed  the  POCs 
that  Rand  had  requested  that  these  units  return  at  least  one  of  the  data 
collection  forms — namely,  the  Unit  Form. 

5.  Letter  dated  5  June  1980,  from  David  Grissmer,  The  Rand 
Corporation,  to  the  Unit  Commanders  of  Sampled  Units, 

SUBJECT:  1979  Reserve  Force  Studies  Surveys:  Suspense  Date. 

This  final  follow-up  letter  was  sent  to  each  sample  unit  (both 
Guard  and  Reserve)  which  had  not  returned  any  surveys  by  May  31,  1980. 
Rand  requested  that  these  units  complete  and  return  at  least  the 
Unit  Form  within  ten  days  of  receipt  of  the  letter.  A  second  copy  of 
the  Unit  Form  was  enclosed  with  this  letter.  An  information  copy  of 
these  letters  was  mailed  to  each  unit's  organizational  superior  (the 
state  adjutant  general  for  Guard  units  and  F0RSC0M  for  Reserve  units.) 
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2100  M  STREET,  N  W..  WASHINGTON,  D  C.  20037,  PHONE  '202  2%  5000 


April  1,  1980 


SUBJECT:  1979  Reserve  Force  Studies  Surveys:  Status  Report 
TO:  Adjutant  General  of 

1.  Thank  you  for  your  continued  cooperation  and  assistance  in  the 
1979  Reserve  Force  Studies  Surveys. 

2.  The  purpose  of  this  letter  is  to  provide  you  with  a  status  report 

on  the  survey  participation  of  sample  units  selected  from  your  state. 

To  date  we  have  received  completed  surveys  from  _  of  _  sample 

units  selected  for  survey  participation.  Survey  materials  have  not 

been  received  from  _  units.  A  list  of  units  in  your  state  which 

have  not  returned  completed  surveys  is  contained  in  Inclosure  1. 

3.  We  appreciate  your  willingness  to  assist  us  in  determining  the 
current  status  of  survey  administration  for  those  units  from 
which  we  have  not  completed  surveys.  If  a  unit  has  not 

had  an  opportunity  to  administer  the  surveys,  we  would  like  to  request 
that  the  unit  complete  the  surveys  during  regular  drill  assemblies 
in  April  or  May.  The  suspense  date  for  returning  survey  materials 
Is  May  31,  1980. 

4.  Should  you  have  any  questions  about  this  survey,  please  call. 
(Commercial:  202-296-5000,  ext.  368).  Thank  you  for  your 
assistance  with  this  important  survey  effort. 

Sincerely, 


Jennifer  A.  Hawes 


Survey  Coordinator 


Enclosure  1 


HOME  OFFICE  THF  KANi>  (  ( )RI’<  >R\ I  ION  I'UH  M  \l\  NIKIII  s\\l\  MONK  \  CAIIIOKN'IA  <>04IX>  I’HONF  J I  l'  I'M-iMII 
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2100  M  STREET.  N  IV  .  WASHINGTON.  O.C.  2005,'  PHONC  (.'0.’  2W-SUOO 


April  2,  1980 


SUBJECT:  1979  Reserve  Force  Studies  Surveys:  Shipment  Verification 
TO:  Unit  Commanders  of  Sampled  Units 


1.  Thank  you  for  your  cooperation  in  the  successful  implementation 
of  Lhc  197^  Reserve  Force  Sttv-ies  Surveys. 

2.  The  purpose,  of  this  letter  ~ ;■  to  acknowledge  receipt  of  your 
unit’s  return  shipment  of  completed  surveys.  The  Defense 
Manpower  Data  Center  received  the  questionnaires  completed 
by  members  of  vour  unit.  Your  unit’s  participation  in  this 
important  survey  will  be  invaluable. 

3.  It  bar.  come  to  our  attention  that  some  units  need  additional 
time  to  complete  the  Commend or  Survey  (Green  Form)  and  Unit 
Survey  (Yellow  Form)  which  were  sent  to  your  unit  in  addition 
to  the  individual,  questionnaires  for  enlisted  unit  members. 
Therefore,  we  ai a  extending  the  suspense  date  for  returning 
all  stuveys  to  May  31,  1980. 

4.  At  this  time,  we.  have  not  received  the  following  questionnaire's) 
from  your  unit: 

(  )  Unit  Survey  Only 

(  )  Commander  Surrey  Only 

(  )  Both  Uni t  &  Commander  Surveys 

Inclosed  with  letter  is  an  extra  copy  of  the  fcrm(s)  whicn  we 
have  not  received  from  your  unit. 

5.  If  you  have  already  returned  the  form(s)  indicated  above,  thank 
yon  (<>r  your  cooperation  in  completing  all  forms  >r  thic  stud;  . 
If  you  have  not  had  an  opportunity  to  complete  them,  we  would 
like  to  request  vour  cooperation  in  completing  the  r cur in mg 
forms)  by  ><:■  y  11 .  1980, 


HOMF  OFFICF  FHF  K  \NI )  (  OKI’OK  MION.  1 7(HI  MM\  SIKIFI  v\St  \  W  ,iC  \  C  \||MV\|'  •■•U  n.  il  >\|  ,1V  V)  1-04  ,  I 
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6.  Should  you  have  any  further  questions  about  this  survey,  please 
contact  the  Survey  Coordinator,  Ms.  Jennifer  Hawes  (Commercial: 
202/296-5000,  ext.  368). 

7.  Your  continued  cooperation  and  assistance  with  this  survey  is 
greatly  appreciated. 


DG/J'net 

Inclosure 

cc:  FORSCOM  -  Ft.  McPherson,  GA 
Adjutant  General  of  _ 


Sincerely, 

David  Grissmer 
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2100  M  STREET,  N  W  UASMINCION  Ll  C  20037.  PHONE  (202  2%-5000 

April  8,  1880 


SUBJECT:  1979  Reserve  Force  Studies  Surveys:  Suspense  Date 

(May  31,  1980) 


TO:  Adjutant  General  Of 


1.  Thank  you  for  your  continued  cooperation  and  assistance  with  the 
1979  Reserve  ^orce  Studies  Surygys . 

2.  It  has  come  to  our  attention  that  several  sample  units  need  addi¬ 
tional  time  to  complete  the  Commander  Survey  (Green  Form)  and 
Unit  Survey  (Yellow  Form)  which,  were  sent  to  units  in  addition  to 
the  individual  questionnaires  for  enlisted  unit  members.  There¬ 
fore,  wv  are  extending  the  suspense  date  for  returning  all  surveys 
to  May  31.  .198(1. 

3.  At  this  time, _ unit(s)  from  you',  state  veiurnad  inroTrolele 

shipments  of  completed  questionnaires  which  did  not  contain  both 
the  Unit  and  Commander  Forms.  As  you  know,  the  Commander  Form 
was  to  be  completed  by  the  Uni „  Commander;  the  Unit  Form  was  to 

be  filled  cut  by  a  unit;  member  des.tgru.ted  by  the  Commander,  usually 
the  Unit  Technician. 


4.  Inclosed  with  this  letter  are  the.  following  materials: 

a.  Attachment  1  which  contains  a  list  of  the  unit(s)  in  your 
state  which  returned  incomplete  shipments  of  completed 
questionnaires,  together  with  a  description  of  the  form(s) 
which  were  not  returned  by  each  unit. 

b.  Attachment  2  which  contains  an  information  copy  of  a 
letter  which  P.and  ’mailed  to  the  unit(s)  in  ycur  state 
which  did  not  return  all  ferns  for  this  survey,  notifying 
thc-ai  of  the  ext  ant  ion  of  the  suspense  date  to  21,  1930. 


5. 


We  will  keep  you  informed  c-f  the  survey  parti 
from  your  state  .  Thanks  again  for  vyo-  c;v  P>- 
successful  imp  lc-mer.  cat  ion  of  the  l-1/*?  Pesory.: 


c i pot’ on  of  unite 
ratio.*  in  t  he 
Fcrco  ft  tiji  ”,  Sur  ye  vs  . 


S: ncerely  , 


j ei’ni  f i.r  A. 


Havo': 
i  mat  PC- 


HOME  OFFICE  THE  RAMI  (OKI’ORATIOS  ITK)  MM\  MKIiI  NAM  \  Mi  M(  A  (  Al  IH  )R\I  \  '.'U-llX.  I’Hi  >\f  Jll.  '.'Hi  a  I 
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Rand 

WASHINGTON  OFFICE 


2100  M  STREET,  N  W  ,  WASHINGTON.  DC  20037,  PHONE  i202i  2%  3000 


June  5,  1980 


SUBJECT:  1979  Reserve  Force  Studies  Surveys:  Updated  Survey  Report 
TO  :  Adjutant  General  of 


1.  Thank  you  for  your  continued  cooperation  and  assistance  with  the 
1979  Reserve  Force  Studies  Surveys . 

2.  The  purpose  of  this  letter  is  to  report  on  the  current  status  of 
the  survey  participation  of  sample  units  from  your  state. 

3.  As  you  were  previously  notified,  we  had  established  a  31  May  1980 
suspense  date  for  returning  all  completed  survey  questionnaires. 

As  of  this  date,  we  have  not  received  a  shipment  from  _  unit(s) 

from  your  state.  A  list  of  units  in  your  state  which  have  not 
returned  questionnaires  at  this  time  is  contained  in  Inclosure  1. 

4.  At  a  minimum,  we  want  to  make  sure  that  we  have  one  of  the  forms 
the  Unit  Form  (YELLOW  FORM)  foT  every  unit  in  our  sample.  We, 
therefore,  sent  a  letter  to  each  non-reporting  unit  to  request 
their  cooperation  in  completing  the  Unit  Form.  An  information 
copy  of  the  letter  which  Rand  sent  to  units  requesting  completion 
of  the  Unit  Form  is  contained  in  Inclosure  2.  Wc  requested  that 
units  complete  the  Unit  Form  within  ten  (10)  days  of  receipt  of 
the  Rand  Letter. 

5.  We  will  continue  to  keep  you  informed  of  the  status  of  this  survey 
effort.  Should  you  have  any  additional  questions,  please  call  ine 
at  Commercial:  202-296-5000,  extension  368. 

Sincerely, 

Jennifer  A.  Hawes 

Survey  Coordinator 


Inclosures  !?) 


HOME  OFFICE  FI  If  RAND  C  ( >RI'OK  \l  H  >\  li»l  \r,IN  SIRUI.  S\NT  \  MONICA  CALIFORNIA  'Hum,.  I’HONt  .JUiWUMIl 
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Rand 

WASHINGTON  OFFICE 

2100  M  STREET  N  W  ,  WASHINGTON.  D  C  20037.  PHONE  1202  .  296-5000 

June  5,  1980 

SUBJECT:  1979  Reserve  Force  Studies  Surveys:  Suspense  Date 

TO:  Unit  Commander  of  Sampled  Units 


1.  Thank  you  for  your  continued  cooperation  and  assistance  with  the 
administration  of  the  1979  Reserve  Force  Studies  Surveys. 

2.  As  you  were  previously  notified,  we  had  established  a  31  May  1980 
suspense  date  for  returning  all  completed  survey  questionnaires. 

As  of  this  date,  we  have  not  received  a  shipment  from  your  unit. 
We  would,  therefore,  appreciate  your  assistance  with  this  survey 
effort  as  described  below. 

3.  At  a  minimum,  we  want  to  make  sure  that  we  have  one  of  the 

forms  —  the  Unit  Form  (YELLOW  FORM)  for  every  unit  in  our  sample. 
An  excra  copy  of  this  form  is  inclosed  with  this  letter.  The 
Unit  Form  should  be  filled  out  by  someone  who  is  familiar  with 
your  unit  records,  usually  the  Unit  Technician. 

a.  If  you  have  not  had  an  opportunity  to  complete  all  forms 
for  your  unit,  we'd  like  your  cooperation  in  completing 
the  inclosed  Unit  Form  at  this  time. 

b.  If  you  have  already  returned  your  shipment,  there  is  a 
strong  possibility  that  the  materials  are  lost  in 
mailing.  Therefore,  we'd  appreciate  your  filling  out 
a  second  Unit  Form.  We're  sorry  for  any  inconvenience 
this  may  cause;  however,  this  data  is  critical  for  the 
success  of  the  survey  effort. 

c.  Please  complete  the  inclosed  Unit  Form  within  ten  (10) 
cavs  of  receipt  of  this  letter  and  return  it  to: 


Ms.  Jennifer  A.  Hawes 
The  Rand  Corporation 
2100  "M"  Street ,  N.W. 
Washington,  D.C.  20037 


HOME  OFFICE:  THE  RAND  CORPORATION,  1700  MAIN  STRUT,  SANTA  MONICA,  (  MlfORNIA  90406  PHONE  (21.T)  391-0411 


June  5,  1980 

Grlssmer  to  Unit  Commanders 
Subject:  Survey  Suspense  Date 
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4.  Should  you  have  any  further  questions  about  this  survey,  please 
call  Ms.  Hawes,  Survey  Coordinator,  Rand  Corporation  at 
Commercial:  202/296-5000,  extension  368. 

5.  Your  unit’s  participation  in  this  important  survey  effort  is 
greatly  appreciated. 


Sincerely, 


Dr.  David  Grissmer 


Inclosure  (Unit  Survey) 

cc:  Adjutant  General  of  Sampled  States 
DCS0PS-F0RSC0M 


73 


REFERENCES 


bailar,  Barbara  A.,  and  C.  Michael  Latiphier ,  Development  of  Survey 
Methods  to  Assess  Survey  Practices,  American  statistical  Associa¬ 
tion,'  Washington;-  P.t. ,  1978. - 

Dillman,  Don  A.,  Mail  and  Telephone  Surveys,  Wiley-Interscience , 

New  York,  1578. - - 

Doering,  Zahava  D.,  David  U.  Grissmer,  and  Jennifer  A,  Hawes,  1979 
Reserve  Force  Studies  Surveys:  Survey  Design,  Sample  DesignT"and 
Adm iri ISt-rdt  1  V"C  Procedures ,  TTTO  Rah d  Corporal  1  Urf,” IT- TT 4  9-M fTAT. j 
August-  fgsr:— - - — — 

Doering,  Zahava  D.,  David  V.  Grissmer,  and  Jennifer  A.  Hawes,  1979 
Reserve  Force  Studies  Surveys:  User's  Manual  and  Codebook.  UTS" 
Rana  corporation ,  n-1755-mRal, August  198  f . 

Heberlein,  Thomas  A.,  and  Robert  Baumgartner.  "Factors  Affecting 
Response  Rates  to  Mailed  Questionnaires:  A  Quantitative  Analysis 
of  the  Published  Literature,"  American  Sociological  Review,  August 
197b,  Vol .  *43 ,  No.  1b,  pp.  4UT-W2. - 

Kanuk,  Leslie,  and  Conrad  Berenson,  "Mail  Surveys  and  Response  Rates: 
A  Literature  Review,"  Journal  of  Marketing  Research,  Vol.  12,  1975, 
pp.  440-453.  - 


